Skip to content

Resolution

This page describes how datarecord reads the layers of a revision as one record. It covers axes, the owner map, crossing rows, what a read returns, flags, materialised revisions and a directory.

A read returns one row at each point, by two rules. Within one layer, datarecord keeps the more specific row, by the broadcast rule. Across layers, a read returns the deeper layer's row where rows of two layers meet.

A read takes the layers through the fold to the owner map The root layer and the child layer feed the fold, which builds the owner map: the layer it assigns each key to. A fold at hand, materialised on disk or kept by the workspace, seeds the fold so that it starts partway. The read joins to the map the files of the layers it lists, and returns each attribute's rows as written. A reader settles each point by the broadcast rule. root layer layers/<root>/ child layer layers/<child>/ owner map a layer per key fold at hand rows as written per attribute reader settles fold seeds read

The step that folds a revision's layers one at a time, root first (the fold), builds the owner map. The map lists, for each attribute, each key a layer wrote and the layer it assigns the key to. A read of an attribute joins to the map the files of the layers it lists, and returns the rows as written. A reader settles each point by the broadcast rule.

Axes

A layer that changes an axis restates it whole, so a revision's axis is the deepest dims/<dim>.parquet on its path from the root. A layer with no file for a dim leaves its parent's axis as it is.

Each axis resolves to the deepest file for it The root layer contains dims/entity.parquet with gas and wind, and dims/scenario.parquet with low and high. The child layer restates the entity axis as gas, wind and solar, and has no scenario file. The resolved entity axis is the child's file and the resolved scenario axis is the root's. root layer dims/entity.parquet entity ctype gas Generator wind Generator dims/scenario.parquet scenario low high child layer dims/entity.parquet entity ctype gas Generator wind Generator solar Generator scenario no file: axis unchanged resolved entity entity ctype gas Generator wind Generator solar Generator scenario scenario low high deepest

The child's entity file is the resolved entity axis, and the root's scenario file is the resolved scenario axis.

  • A multi-dim's axis resolves the same way, from its file keyed by its struct column.
  • The labels of an axis come in the deepest file's row order, the axis order. A label added again after its removal sits where the file that adds it puts it.
  • A property is a column of its subject's axis file, so its values change only when a layer restates the axis.
  • A dim with one_of has no file, and its axis is the schema's list at every revision.

The owner map

The owner map is the fold's result: per attribute, each key a layer wrote and the layer the map assigns it to. A key is the attribute's dims in partial and the attribute's name. The map contains no values, so its size does not depend on the length of a series or a curve.

Column Contains
one per dim in partial the key's label, or NULL where the row leaves the dim NULL. A multi-dim is one struct column
attribute the attribute's name
layer_uuid the layer the map assigns the key to
varies a struct with a boolean per dim: whether the layer's rows of this key set the dim
broadcast a struct with a boolean per dim: whether the layer's rows of this key leave the dim NULL
breakpoints whether the layer's rows of this key form a curve

The map has each key as its layer wrote it, a NULL included. It has one row per layer and distinct key, and one overlap row for each pair of rows that cross.

Folding a layer onto its parent

The fold takes the parent's map and the layer's own keys, and returns the layer's map. Three terms describe how the rows of two layers meet:

Term Of two rows, means
sets a row sets a dim in partial where its key has a label, and any other dim of the attribute where its rows vary along it
covers on each dim in partial, the two rows have the same label, or one of them has NULL
crosses the two rows cover a common point, and each row leaves NULL a dim that the other row sets

The fold runs three steps, in this order:

  1. It drops each parent row that has a label or tuple that the layer removes, on a dim in partial. A row that leaves the dim NULL stays.
  2. It drops each parent row that a row of the layer covers, where the layer's row sets every dim that the parent row sets.
  3. It adds the layer's rows, and one overlap row for each parent row that a row of the layer crosses.

Take partial = {"entity", "scenario"} and a root that sets p_nom for gas, wind and coal. Its child removes coal, sets gas to 60 and then sets 999 for the scenario high:

One layer folds onto its parent's owner map With partial dims entity and scenario, the parent's owner map contains the root's rows gas NULL, wind NULL and coal NULL. The child removes coal and writes gas NULL 60, NULL high 999 and gas high 999. The fold drops the root's coal row, which has the removed label, and the root's gas row, which the child's gas NULL covers. It adds the child's three rows and one overlap row, wind high, which the owner map assigns to the child, where the root's wind NULL crosses the child's NULL high. parent map p_nom entity scenario layer gas NULL root covered wind NULL root coal NULL root removed child layer p_nom, removes coal entity scenario value gas NULL 60 NULL high 999 gas high 999 owner map after the fold entity scenario layer wind NULL root gas NULL child NULL high child gas high child wind high child overlap fold drop removed, drop covered, add rows and overlaps root layer child layer overlap row dropped by the fold

The fold drops the root's coal row, which has the removed label, and the root's gas row, which the child's gas row covers. It keeps the root's wind row and adds the overlap row (wind, high), which the owner map assigns to the child. The child's own (gas, high) is a row that its WorkingRecord staged, because the second set crosses the first.

A parent row that sets fewer dims than a covering row of the layer stays as written. A child that writes only (gas, high) = 70 leaves the root's (gas, NULL) in the map. By the broadcast rule, a read takes (gas, high) from the child.

Crossing rows

Two rows of two layers cross where they cover a common point and each row leaves NULL a dim that the other row sets. At such a point neither row is more specific, so the broadcast rule alone does not settle it. The fold therefore adds an overlap row, which the owner map assigns to the deeper layer, with every label that either row sets.

The fold adds an overlap row where rows of two layers cross With partial dims entity and scenario, the root writes p_nom for gas and wind with scenario NULL, and the child writes p_nom 999 for scenario high with entity NULL. The owner map has these three rows and two overlap rows that it assigns to the child, gas high and wind high. The read returns five rows: gas NULL 50, wind NULL 100, NULL high 999, gas high 999 and wind high 999. root layer p_nom entity scenario value gas NULL 50 wind NULL 100 child layer p_nom entity scenario value NULL high 999 owner map entity scenario layer gas NULL root wind NULL root NULL high child gas high child overlap wind high child overlap read p_nom entity scenario value gas NULL 50 wind NULL 100 NULL high 999 gas high 999 wind high 999 fold read root layer child layer overlap row

The child's (NULL, high) crosses both root rows, so the map gains (gas, high) and (wind, high), which it assigns to the child. At (wind, high) the overlap row sets both dims, so by the broadcast rule the value there is 999. The crossed parent rows stay as written, and (wind, low) is 100.

A later fold treats an overlap row as any other row, which it may drop, cross or remove.

Where a row applies

A NULL on a dim in partial means every label of that dim's axis as resolved at the revision read. A row therefore applies to a label added below the revision that wrote it, or one staged in a WorkingRecord. In the example above, a child that adds the scenario mid has (gas, mid) at 50.

A row applies at every point that no more specific row covers, whichever layer wrote it. So a read returns a deeper layer's value only at the points its rows apply to. With p_max_pu over entity, scenario and snapshot, a root row (gas, NULL, 1) = 1.0 and a child row (NULL, high, 0) = 2.0 leave (gas, high, 1) at 1.0, from the root.

Reading an attribute

record.attributes[name] returns the attribute's rows from the layers that the map assigns a key to, as written. A key left NULL stays NULL. Where the map has an overlap row, the read also returns a value at the overlap row's labels. That value comes from the layer the map assigns the overlap row to. The read expands no NULL, settles no point, and returns rows in no fixed order. An attribute that no layer wrote is an empty relation, typed as declared, and the reader applies the schema's default.

A layer may contain rows of one attribute that set different dims in partial. The child above is one: its (gas, NULL), (NULL, high) and (gas, high) all apply at (gas, high). At such a point the read keeps the layer's row that leaves no dim NULL that another row there sets. Where no row does, the read fails, naming the attribute and the point. A curve is kept or dropped whole.

See also

The record page describes RecordLike and settling rows. Why the owner map contains keys as written gives the reasons for the map's shape.

Labels outside the key

A dim outside partial is in no key, so the fold reads none of its labels. The read drops each row that has a removed label on such a dim, or a removed tuple of such a multi-dim. A removed label is one that a layer's restatement dropped and the revision's axis lacks.

Take partial = {"entity"}, a root with p_max_pu for wind at snapshots 0, 1 and 2, and a child that removes snapshot 2:

A read drops a row with a removed label on a dim outside partial With partial entity alone, the root writes p_max_pu for wind at snapshots 0, 1 and 2 with 0.3, 0.8 and 0.5, and the snapshot axis 0, 1, 2. The child restates the snapshot axis as 0 and 1, which removes snapshot 2, and writes no p_max_pu file. snapshot is in no key, so the owner map assigns wind to the root. The read returns the root's rows at snapshots 0 and 1 and drops the row at snapshot 2. root layer p_max_pu entity snapshot value wind 0 0.3 wind 1 0.8 wind 2 0.5 read p_max_pu entity snapshot value wind 0 0.3 wind 1 0.8 root layer snapshot axis snapshot 0 1 2 child layer snapshot axis snapshot 0 1 read keeps the root's rows, drops snapshot 2 restate without 2 no p_max_pu file root layer child layer read removed label, dropped row

The child writes no p_max_pu file, yet the read drops the root's row at snapshot 2 and returns its other two rows.

With connection outside partial, a connection attribute's key is the attribute's name alone. Removing a tuple then removes its values at read, as it does for a multi-dim in partial at the fold.

A dim outside partial

A crossing along a dim outside partial has no key column for an overlap row to set. A WorkingRecord settles such a crossing as it stages. Where a staged row leaves the dim NULL over a parent row that sets it, the commit also writes the value along the parent's labels.

A child that leaves a dim outside partial NULL is written along the parent's labels With partial dims entity alone, the root writes p_max_pu over snapshot for every entity: NULL 0 0.2 and NULL 1 0.5. The edit sets wind to 0.9 with snapshot NULL. store.commit writes the child layer as wind NULL, wind 0 and wind 1, all 0.9, so the child's rows set every dim that the root's rows set. The fold refuses a layer with the first row alone. root layer p_max_pu entity snapshot value NULL 0 0.2 NULL 1 0.5 edit set(..., entity="wind") entity snapshot value wind NULL 0.9 child layer as written entity snapshot value wind NULL 0.9 wind 0 0.9 wind 1 0.9 read wind snapshot value 0 0.9 1 0.9 commit A layer with the first row alone: the fold refuses it.

With partial = {"entity"}, the child's rows at wind set every dim that the root's rows set. Nothing crosses, and wind is 0.9 at each snapshot. The fold refuses a layer written another way that leaves such a crossing. It checks the layer when a workspace first folds it:

ValueError: layer 0f74b479-422a-4364-b29f-36758ea3105c leaves 'snapshot' NULL for 'p_max_pu' at {'entity': 'wind'} where a parent row names it; write it along the parent's 'snapshot' labels there, as `WorkingRecord.set` does

A wider schema

The fold reads each layer under the schema in its own manifest. A child's schema may be wider than its parent's where the change passes compatible_with. The fold then extends the parent's map to the child's schema before it folds the child.

  • Where a dim is added to an attribute's dims, the parent's rows of that attribute broadcast along it.
  • Where a dim is added to partial, the fold splits each parent row that varies along it by the labels that its layer wrote there.
The fold splits the parent's rows by their labels when a dim joins partial The root, under partial entity, writes gas low 1 and gas high 2. The child, under partial entity and scenario, writes gas high 3. The fold splits the root's map row by the scenarios the root wrote: gas low and gas high, both assigned to the root. The child's gas high covers the second, which the fold drops. The read gives gas low 1 and gas high 3. root layer partial {entity} entity scenario value gas low 1 gas high 2 child layer partial {entity, scenario} entity scenario value gas high 3 parent map, split entity scenario layer gas low root gas high root covered read p_nom entity scenario value gas low 1 gas high 3 split fold dropped by the fold

Under partial = {"entity"}, the owner map assigns gas's scenarios whole to the root. Under the child's partial = {"entity", "scenario"}, the fold splits the root's row into (gas, low) and (gas, high), and the child's (gas, high) covers the second. The read gives (gas, low) 1 and (gas, high) 3.

Flags

The fold computes varies, broadcast and breakpoints for each row of the map from the layer's rows of that key. An overlap row takes the flags of the layer row it comes from. record.flags() unions them per attribute:

flags() unions the owner map's flag fields per attribute The owner map contains p_max_pu for wind, which sets snapshot, p_max_pu for gas, constant over snapshot, and p_nom with entity NULL. flags() takes the union per attribute: p_max_pu varies along entity and snapshot and broadcasts along snapshot, and p_nom broadcasts along entity and scenario. owner map flag fields set to true attribute entity varies broadcast p_max_pu wind entity snapshot - p_max_pu gas entity snapshot p_nom NULL - entity scenario flags() attribute varies broadcast p_max_pu entity snapshot snapshot p_nom - entity scenario

p_max_pu has snapshot in both sets, because wind's rows set it and gas's row leaves it NULL. p_nom broadcasts along entity and scenario, the two dims it is over.

  • flags(**categories) unions the flags of the rows that apply to the labels of those categories. A row NULL on the dim applies to every label.
  • Each struct has a field for every dim the schema declares. A dim that a multi-dim draws on has its own field, so flags() reports bus as broadcast for a marginal_price constant over buses.
  • The flags describe the rows in the map. So a NULL row that deeper rows cover at every label still puts its dim in broadcast.

See also

The record page describes what a reader does with flags.

Materialised revisions

To materialise a revision is to write its owner map, pointers to the files of its axes and its removed labels under layers/<id>/resolved/, with Revision.materialise(). A read of a descendant then stops at that revision and reads no layer above it.

A fold starts from the deepest proper ancestor whose fold is at hand, either materialised on disk or kept as tables by the workspace.

A fold starts from the nearest fold at hand The ancestry runs root, a, b, c, d, child. Revision a is materialised on disk, and the workspace keeps the fold of c as tables. For a record opened on child, the fold starts from c, the deepest fold at hand, and folds d and then child onto it. Root, a and b are not read. root not read a materialised b not read c fold held d parent layer child child layer the fold: seeded from c, then folds d, then child parent to child

For a record opened on child, the fold starts from c, the deepest fold at hand, and folds d and then child onto it. It reads nothing of root, a or b.

  • A read gives the same answers after you materialise, and opens fewer layers.
  • datarecord never starts a revision's fold from that revision's own cache.
  • A materialised map has keys as written, so a NULL in it applies to a label that a descendant adds.
  • A read under a wider schema extends a materialised map as it extends one folded live.

See also

The format page lists the files under resolved/. Branch and materialise shows when to call materialise().

A directory

Workspace.at(uri) reads a standalone directory as a fold over one layer, whose layout is a layer's.

Reading a directory as a fold over one layer The directory out contains manifest.json, dims and attributes, the layout of one layer. Read with ws.at, it is a fold over that one layer, so the owner map assigns every key to that layer. out/ manifest.json, dims/, attributes/ owner map entity scenario layer gas NULL out wind NULL out ws.at("out/") a fold over one layer

The owner map assigns every key to the one layer, the axis order is each file's row order, and partial has no effect. One Record reads both a directory and a revision, so a reader cannot tell them apart. store.layer(id) reads one layer of a store alone in the same way.

See also

One interface over a directory and a tree gives the reasons for one Record. The store page describes Workspace.at and Workspace.write.