Store¶
Workspace
¶
Workspace(
database: str = ":memory:",
*,
con: DuckDBPyConnection | None = None,
name: str = "datarecord",
)
A DuckDB database and the files it reaches, shared by every store on it.
con is the calling thread's cursor: one workspace serves every thread, and
their work shares one database - its tables, never its relations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
database
|
str
|
DuckDB database, |
':memory:'
|
con
|
DuckDBPyConnection | None
|
A connection to attach |
None
|
name
|
str
|
What |
'datarecord'
|
Raises:
| Type | Description |
|---|---|
BinderException
|
If |
Source code in src/datarecord/store/store.py
con
property
¶
The calling thread's cursor, made on its first use in that thread.
qualified
¶
table in the main schema of the workspace's database, named in full.
Notes
Source code in src/datarecord/store/store.py
unregister
¶
Drop the schema name a record registered, with its views and its registered() row.
A closed WorkingRecord's staging tables go with its last registration.
Raises:
| Type | Description |
|---|---|
CatalogException
|
If |
TransactionException
|
Where the cursor already holds an open transaction. |
Notes
Source code in src/datarecord/store/store.py
registered
¶
Every registered name, with the revision it reads, whether it is working, the schemas it held, and when it was created.
Notes
Source code in src/datarecord/store/store.py
process
classmethod
¶
process() -> Workspace
The process's workspace, created on first use on :memory:.
What a deserialised Revision or Claim attaches to, having no
workspace passed; nothing else reaches it unless named.
Source code in src/datarecord/store/store.py
files
¶
The Files for uri's scheme: local without ://, remote for s3://.
Raises:
| Type | Description |
|---|---|
ValueError
|
For any other scheme. |
Source code in src/datarecord/store/store.py
store
¶
store(uri: str) -> Store
This workspace's store at uri, one object per normalised location.
Raises:
| Type | Description |
|---|---|
ValueError
|
For an empty |
Source code in src/datarecord/store/store.py
at
¶
at(uri: str) -> Record
The standalone record at uri, read under its own manifest.json.
A local uri is resolved, as store resolves one, so a later chdir
cannot point it at another record.
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If |
ValueError
|
If the manifest names a parent: a layer of a tree is read alone
through its store's |
Notes
Source code in src/datarecord/store/store.py
write
¶
write(
source: LayerData | RecordLike,
uri: str,
*,
meta: Meta | None = None,
) -> Record
Write source as a standalone record at uri, and read it back.
A standalone record is one layer at its own URI, written to a URI that
does not exist yet: its data first, then its manifest, with a fresh id
and no parent, so one whose write failed reads as no record and one
written again elsewhere is never read as the one before it. A failed
write leaves its data files at uri, locally as remotely, for the
caller to delete before writing there again.
Raises:
| Type | Description |
|---|---|
FileExistsError
|
If anything exists at |
Source code in src/datarecord/store/store.py
close
¶
Close every thread's cursor and detach the database.
The connection is closed with them only where the workspace opened it; one passed in stays open for its owner.
Source code in src/datarecord/store/store.py
Store
¶
Store(uri: str, workspace: Workspace)
One store location: its layers, each carrying the manifest that makes it a revision.
Holds no connection of its own: every read and write goes through its workspace, which several stores share.
Attributes:
| Name | Type | Description |
|---|---|---|
uri |
The location: absolute, never empty. |
Notes
Source code in src/datarecord/store/store.py
layer_dir
¶
Revision id's layer directory, with a trailing slash.
Notes
Source code in src/datarecord/store/store.py
manifest_uri
¶
claim_uri
¶
axis_uri
¶
Where revision id's layer holds dim's axis.
Notes
Source code in src/datarecord/store/store.py
resolved_dir
¶
Revision id's node caches, under its layer where no manifest lists them.
Notes
Source code in src/datarecord/store/store.py
owner_map_uri
¶
Where revision id's owner map is materialised.
Notes
Source code in src/datarecord/store/store.py
dim_owners_uri
¶
Where revision id's dim owners are materialised.
Notes
Source code in src/datarecord/store/store.py
claim
¶
Hand out a new layer id under parent, by its create-only claim.json.
The layer is pending, reached by no read, until Claim.commit writes
its manifest; meta is the revision's, which the commit may extend
but not change.
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValidationError
|
If |
Source code in src/datarecord/store/store.py
commit
¶
commit(
source: WorkingRecord, *, meta: Meta | None = None
) -> Revision
commit(
source: LayerData | RecordLike,
*,
parent: Revision | None,
meta: Meta | None = None,
) -> Revision
commit(
source: LayerData | RecordLike,
*,
parent: Revision | None | _Base = OF_WORKING_RECORD,
meta: Meta | None = None,
) -> Revision
Write source as a new revision under parent: a claim and its commit in one.
A WorkingRecord brings its parent, the revision it was built over,
and is written as its edits, a delta against that revision. Any other
source names its parent, None for a new root, and is written
whole.
Raises:
| Type | Description |
|---|---|
TypeError
|
If a |
ValueError
|
If |
Notes
Source code in src/datarecord/store/store.py
revision
¶
revision(id: UUID) -> Revision
Look one revision up, reading its manifest and any parents' not yet held.
Raises:
| Type | Description |
|---|---|
KeyError
|
If this store has no revision |
Notes
Source code in src/datarecord/store/store.py
revisions
¶
revisions(*, meta: Meta | None = None) -> list[Revision]
Every revision in this store whose meta holds each pair of meta, unordered.
A pending layer is none. One listing of layers/, reading only the
manifests the workspace does not already hold; a value matches only
one of its own type, so 1 matches neither 1.0 nor True.
Raises:
| Type | Description |
|---|---|
ValidationError
|
If |
Notes
Source code in src/datarecord/store/store.py
claims
¶
claims() -> list[Claim]
Every pending layer in this store, claimed and not committed, oldest first.
Read from the claim files on each call and never cached, since a pending layer may commit at any time.
Notes
Source code in src/datarecord/store/store.py
cleanup
¶
Delete pending layers: those in ids, or those claimed older_than ago or earlier.
Only a pending layer is deleted: each layer's manifest is rechecked right before its delete, without a lock. A job may hold its claim for days, so nothing deletes but this call, and a layer whose job still runs must be neither named nor young enough to match.
Returns:
| Type | Description |
|---|---|
list of UUID
|
The ids of the layers deleted. |
Raises:
| Type | Description |
|---|---|
TypeError
|
Unless exactly one of |
ValueError
|
Naming each of |
Notes
Source code in src/datarecord/store/store.py
layer
¶
layer(id: UUID) -> Record
Revision id's layer read alone, as the one layer it is.
Under its own manifest, as Workspace.at reads a standalone directory.
Not the revision's record - nothing beneath it is folded in.
Raises:
| Type | Description |
|---|---|
KeyError
|
If this store has no revision |
Source code in src/datarecord/store/store.py
Claim
¶
Bases: _OfStore
A layer id handed out by Store.claim, whose layer is pending until commit.
Serialises as its store's location and its id, and validates or unpickles
back through Workspace.process() by reading its claim file, so a job in
another process commits it.
Attributes:
| Name | Type | Description |
|---|---|---|
parent |
UUID | None
|
The revision the layer will fold over, |
meta |
Meta
|
The revision's meta as claimed, which the commit keeps. |
created |
datetime
|
When the id was handed out. |
commit
¶
commit(
source: LayerData | RecordLike,
*,
meta: Meta | None = None,
) -> Revision
Write source as this claim's layer, then its manifest, which makes it a revision.
The data goes straight into the layer directory; the manifest, written
create-only and last, lists its files, its full schema, the claim's
parent and the claim's meta with meta added. A WorkingRecord is
written as its edits.
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
ValueError
|
If |
PreconditionFailedError
|
If another commit of this claim wrote the manifest first. |
KeyError
|
If the claim's parent is gone from the store. |
ValidationError
|
If |
Source code in src/datarecord/store/store.py
1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 | |
Revision
¶
Bases: _OfStore
A node in the tree of layers.
Each revision adds one parquet layer on top of its parent's; the record it
resolves to is that layer over its ancestors'. The layer location derives
from id via its store's layer_dir, it is not stored.
Serialises its store as the location, and validates or unpickles back
through Workspace.process(), reading the manifest and every parent's
the process does not yet hold - so ancestry() answers in a fresh process.
Nothing cached travels.
record
property
¶
record: Record
This revision's resolved view, as a Record built once from its ancestry.
The ancestry is truncated at the deepest materialised ancestor, so a
deep tree resolves from few entries. Cached rather than rebuilt per
call: layers are write-once, and a record built before materialise
reads the same answers over the longer ancestry. The
Record holds no cursor, so the cache serves every thread. Once it is
closed, the next call builds and caches another.
manifest
property
¶
manifest: Manifest
This revision's manifest: its parent, schema, files and meta.
Notes
materialise
¶
Write this node's caches - owner map, dim owners and dead labels.
A policy rather than a lifecycle step, and purely additive: it changes no answer, only how many layers a descendant's read touches. Once these exist, a read stops here instead of walking further up.
Notes
Source code in src/datarecord/store/store.py
ancestry
¶
Revision ids along the root->self path, root first.
Notes
Source code in src/datarecord/store/store.py
Manifest
¶
A layer's manifest.json: written last and create-only, it makes the revision exist.
Attributes:
| Name | Type | Description |
|---|---|---|
parent |
UUID | None
|
The revision this layer folds over, |
schema |
InstanceOf[Schema]
|
The full schema the layer was written under, never a delta. Taken as
built where it is handed in, and parsed by |
meta |
Meta
|
The revision's own: set at the write, and not inherited by a child,
unlike |
Notes
write
¶
Write this manifest at uri, create-only.
Raises:
| Type | Description |
|---|---|
PreconditionFailedError
|
If a manifest is already there. |
read
classmethod
¶
The manifest at uri, or None where there is none.
Its schema is taken from schemas by schema_digest, and parsed and
added there only where it is missing.
Raises:
| Type | Description |
|---|---|
ValidationError
|
If the manifest, or a schema not in |