Record¶
Record
dataclass
¶
Record(
revision_id: UUID,
sources: list[LayerSource],
workspace: Workspace,
)
A record's layers folded last-writer-wins, read as DuckDB relations.
One class for every backing, because a fold over one source degenerates to
a scan of it - so a plain parquet directory is read as the one layer it is
(Workspace.at) rather than by a second code path. RecordLike is the
protocol a caller annotates against; this is the class a caller holds.
Nothing up to the last frozen source can go stale, so the fold over them is
kept in shared tables. A source after it is folded onto those tables per
read, so a staged edit reads back with nothing to invalidate. The record
holds those tables from construction until close(), or until it and
every relation taken from it are collected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
revision_id
|
UUID
|
The node whose frozen layers are folded, which names the shared tables their fold is kept in. |
required |
sources
|
list of LayerSource
|
Root first, ending in the layer being resolved, truncated at the
deepest materialised ancestor ( |
required |
workspace
|
Workspace
|
Where the calling thread's cursor comes from. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
On a read once the record is |
Notes
con
property
¶
The cursor of the calling thread, which the relations of this record are planned on.
Notes
frozen
property
¶
Whether every source is frozen, so a fold over them cannot go stale.
False exactly when the last source is a staging area.
close
¶
Release the tables this record reads, each dropped once no other record or registration holds it.
Closing again does nothing. A closed record refuses every read with a
ValueError, and a relation taken from it before fails once its
tables are dropped.
Notes
Source code in src/datarecord/resolve/record.py
dims
¶
dims() -> LazyRelations
Every dim the schema declares, an empty axis where no layer holds one.
properties
¶
properties() -> LazyRelations
Every subject the schema declares properties of, an empty axis where no layer holds one.
Source code in src/datarecord/resolve/record.py
attributes
¶
attributes() -> LazyRelations
Every attribute the schema declares, an empty relation where no layer holds a row.
Source code in src/datarecord/resolve/record.py
flags
¶
flags(
**categories: str | Sequence[str],
) -> dict[str, Flags]
Straight off the owner map, which folded these in for free.
Notes
Source code in src/datarecord/resolve/record.py
select
¶
select(**labels: Any) -> Selection
This record at the labels named, as datarecord.select takes them.
Raises:
| Type | Description |
|---|---|
ValueError
|
As |
Notes
Source code in src/datarecord/resolve/record.py
register
¶
Create the schema name with a view per dim, property subject and attribute.
The views sit in name.dims, name.properties and name.attributes of
the workspace's database, as in datarecord.name.dims.bus, each reading as
its relation here; over a WorkingRecord they follow every edit. They hold
the tables the record reads, its folds and a WorkingRecord's staging
tables, until unregister(name), whether the record is closed or not.
name is listed in Workspace.registered().
Raises:
| Type | Description |
|---|---|
CatalogException
|
If the schema |
TransactionException
|
Where the cursor already holds an open transaction, or another
cursor registers |
Notes
Source code in src/datarecord/resolve/record.py
RecordLike
¶
Bases: Protocol
What a record answers, however it is backed, as DuckDB relations.
Read-only: writing is store.commit(source, parent=...) or
ws.write(source, uri), whose writer reads a LayerData rather than this -
a framework's own RecordLike reaches it through an adapter.
schema
property
¶
schema: Schema
The record's schema: its dims, its attributes, its patch granularity.
Notes
con
property
¶
The connection every relation here is planned on, the calling thread's.
dims
property
¶
dims: Relations
Axis relations keyed by dim, each holding only the dim's label column.
A multi-dim's label column is its struct column, a field per
coordinate. Properties are in properties, attributes in
attributes.
properties
property
¶
properties: Relations
Per dim declaring properties, its key and their stored values, no default applied.
Notes
attributes
property
¶
attributes: Relations
Long attribute relations, keyed by attribute name - one per file.
A row carries its coordinates alone; select by category through
properties.
Notes
flags
¶
flags(
**categories: str | Sequence[str],
) -> dict[str, Flags]
Every attribute with rows, mapped to the shape its rows take.
Given categories of one dim, only rows reaching a label among them
count, and only attributes those labels carry.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the keywords categorize no dim or several. |
Notes
Source code in src/datarecord/protocols.py
Relations
module-attribute
¶
What a Record hands over: named relations, each an unmaterialised plan.
The Mapping ABC, so a plain dict satisfies it as fully as LazyRelations
does.
Notes
LazyRelations
¶
Bases: Mapping[str, DuckDBPyRelation]
A Relations whose values are built on __getitem__, not up front.
For a backing where building a relation is itself I/O: read_parquet
reads the footer to bind the schema, a round trip per file against a remote
record.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
keys
|
tuple[str, ...]
|
Every key this mapping holds, in iteration order. Listing them must be cheap: nothing here builds a relation. |
required |
build
|
Callable[[str], DuckDBPyRelation]
|
Called with one key to produce its relation, once per |
required |
Notes
Source code in src/datarecord/protocols.py
Flags
dataclass
¶
Which axes an attribute's rows actually use.
The two sets are not complements: a dim in both means the rows
disagree, which is the instruction to use both containers rather
than an ambiguity to resolve. varies | broadcast is the test for
whether an attribute touches a dim at all.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
varies
|
frozenset[str]
|
Dims some row of this attribute sets. |
required |
broadcast
|
frozenset[str]
|
Dims some row leaves NULL, i.e. "all values of that dim". |
required |
breakpoints
|
bool
|
Whether any row carries a breakpoint. Not a dim. |
False
|
select
¶
select(record: RecordLike, /, **labels: Any) -> Selection
record at the labels named, as xarray's sel takes them.
A keyword names a dim; one naming a multi-dim's coordinate that is no dim
narrows its tuples, and any other names a property by its bare name and
narrows its subject to the labels whose value, its default applied, is
selected. One label squeezes its dim. A list, a slice (both ends
inclusive, None open) or a partial date string on a datetime axis keeps
it. Every label referring to a dropped one goes with it, through
properties and multi-dim tuples, until nothing more goes. Over a
Selection, its keywords are merged and its base selected.
Returns:
| Type | Description |
|---|---|
Selection
|
The view, which reads the axes it matched and propagated on as they were at the call, and plans every attribute on the base's connection. |
Raises:
| Type | Description |
|---|---|
ValueError
|
Naming the keyword: one naming no dim, coordinate or property, a multi-dim, a coordinate of several multi-dims or a property several subjects declare; a label the axis lacks; an empty list; a slice with a step, of an axis neither numeric nor datetime, with an end the axis cannot compare or covering no label; a squeezed label another keyword's selection drops; a squeeze folding a category into the default a kept label NULL on it does not take; squeezing every dim of a record with attributes. |
Source code in src/datarecord/select.py
238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 | |
Selection
dataclass
¶
Selection(
base: RecordLike,
labels: Mapping[str, Any],
schema: Schema,
_squeezed: Mapping[str, Any],
_kept: Mapping[str, DuckDBPyRelation],
)
A record at the labels select names, itself a RecordLike.
A dim given one label is squeezed out of schema; every other dim keeps
the labels that refer to no label the selection drops. dims and
properties keep the base's order, and every relation is a plan on the
base's connection.
Attributes:
| Name | Type | Description |
|---|---|---|
base |
RecordLike
|
The record selected from, never a |
labels |
Mapping[str, Any]
|
The keywords asked for, normalized: one label for a squeezed dim, else the list of labels matched. |
schema |
Schema
|
The base's schema without the squeezed dims, their axes, their properties and every property referring to one. |
Notes
con
property
¶
The base's connection, which every relation here is planned on.
dims
property
¶
dims: LazyRelations
The axes of the dims left, each at its kept labels in axis order.
properties
property
¶
properties: LazyRelations
The stored properties left, at the kept labels of their subject.
attributes
property
¶
attributes: LazyRelations
Long rows at the kept labels or NULL there, each squeezed dim settled and dropped.
Notes
flags
¶
flags(
**categories: str | Sequence[str],
) -> dict[str, Flags]
The base's flags without the squeezed dims, overstating the view's rows.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the keywords categorize no dim of the view, or several. |
Notes
Source code in src/datarecord/select.py
select
¶
select(**labels: Any) -> Selection
Select base once more, labels replacing the keywords of the same name.
Raises:
| Type | Description |
|---|---|
ValueError
|
As |
lift
¶
rel, rows of attribute over this view, with the squeezed labels put back.
Each squeezed dim the base declares attribute over is added as a
column at its label, and each squeezed coordinate back into its
multi-dim's struct; a NULL struct stays NULL. An attribute the base
declares over no squeezed dim comes back as it came.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the base declares no |
Notes
Source code in src/datarecord/select.py
settle
¶
settle(
rel: DuckDBPyRelation,
labels: Mapping[str, DuckDBPyRelation],
flags: Flags,
*,
filter: bool = True,
) -> DuckDBPyRelation
Settle the long rows of rel by the broadcast rule, one row per coordinate.
A row NULL on a coordinate in labels takes each label there, and a row
naming a label outside labels, or reaching none, is dropped. At each
coordinate the row naming every coordinate in labels that the other rows
there name is kept. A coordinate outside labels is kept as written, a
NULL there being one more label, and breakpoint is no coordinate, so a
curve is kept or dropped whole.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
labels
|
Mapping[str, DuckDBPyRelation]
|
Per coordinate, a relation holding its labels, or its tuples for a multi-dim, in a column of the coordinate's name. |
required |
flags
|
Flags
|
The attribute's |
required |
filter
|
bool
|
False keeps a row naming a label outside |
True
|
Raises:
| Type | Description |
|---|---|
Error
|
On fetch, where no row names every coordinate the rows reaching it name. |