DEP 9: a Workspace holds the connection, a Store holds a location¶
| DEP | 9 |
|---|---|
| Title | a Workspace holds the connection, a Store holds a location |
| Author | Jonas Hörsch |
| Status | Implemented (2026-10-10) |
| Created | 2026-10-09 |
| Version | Unreleased |
The design landed in Workspace and Store, layers, WorkingRecord, tools and the package. Where they and this DEP differ, follow those pages.
Abstract¶
duck's module globals split into two objects. A Workspace has the DuckDB connection and file access, local or remote, which is all a standalone directory needs. A Store has one location, with its manifest, revision tree and layers, and works through a workspace that several stores share. A revision crosses a process boundary as its store's location.
Motivation¶
Checked against 49b7268:
default_connection()is a process singleton on:memory:, and therevisionstable lives only there. A revision deserialised in a new process has an empty ancestry.- Only the code that reads the schema calls
base_uri_of(con). Layers and caches useDEFAULT_BASE_URI, soconnect(base_uri=X)writes its manifest underXand its layers elsewhere. With an empty default, datarecord writeslayers/<id>/into the working directory. PyPSA.results(n)builds relations on DuckDB's default connection, so the next union fails.- Revisions cannot be listed.
- A remote manifest is unreadable and unwritable, and a failed remote write leaves part of a layer.
- Of eight threads on one DuckDB 1.5.5 connection, seven fail.
materialise()writes theinputsowner map first, so a reader can fold from half a cache.
One object holding location and connection would fix these, but a standalone directory would then carry a location it never uses.
Specification¶
class Workspace:
con: DuckDBPyConnection
def __init__(self, database: str = ":memory:") -> None: ...
@classmethod
def process(cls) -> Workspace: ...
def store(self, uri: str) -> Store: ...
def at(self, uri: str) -> Record: ...
def write(self, source: LayerData | RecordLike, uri: str) -> Record: ...
def close(self) -> None: ...
class Store:
uri: str
workspace: Workspace
def root(self, source: LayerData | RecordLike) -> Revision: ...
def revision(self, id: UUID) -> Revision: ...
def revisions(self) -> list[Revision]: ...
def layer(self, id: UUID) -> Record: ...
def layer_dir(self, id: UUID) -> str: ...
def resolved_dir(self, id: UUID) -> str: ...
class Revision:
def child(self, source: LayerData | RecordLike) -> Revision: ...
- Only
ws.store,ws.at,ws.writeand a tool'sconname the workspace. Everything else takes it from its source. ws.store(uri)returns one store per normalised location, filled withdict.setdefault.ws.store("")raisesValueError, and a local path becomes absolute. A store has no connection and noclose.ws.at(uri)requires the directory's ownmanifest.json. A tree layer reads throughstore.layer(id).- A
Recordkeeps exposingcon, the calling thread's cursor, so that a tool can run DuckDB SQL. - There is no empty revision.
store.rootandrev.childwrite the layer, then widen the manifest, then writerevisions/<id>.json, create-only. The layer counts as written once its revision file exists, and a failed or incompatible layer leaves the manifest untouched. ws.writewrites the manifest last. Before anyCOPY, a remotews.writerefuses a URI under whichFiles.listfinds anything, becauseCOPYwould overwrite an existing record's files before theAbsentmanifest write fails.materialise()writesowner_map/inputs.parquetlast, and each local file goes to a temporary name and is renamed.WorkingRecord(record)takes noconand has nocommit.w.editsis its stagedLayerData.rev.childraises when a staged source's base is another revision. AWorkingRecordthat still has its edits after a write reads as before.- The tree is
<location>/revisions/<id>.json,{"id", "parent"}, with a UUID drawn in Python. The workspace'srevisionstable,(location, id, parent), caches those files and is only added to.revisions()returns every revision, unordered.revision(id)on a miss reads the file and each uncached parent, and raisesKeyErrorwhere the first file is absent. A concurrent duplicate insert'sTransactionExceptionis caught. - Parquet goes through DuckDB. The manifest, revision files, listings and existence checks go through
Files, withread_text,write_text(..., expect=Version | Absent | None),listandexists. s3://is remote and a URI without://is local. Any other scheme raises. LocalFilesusespathlib,open(..., "x")forAbsent, and a SHA-256Versionchecked under a sibling.lock. RemoteFilesuses boto3 behinddatarecord[remote] = ["boto3>=1.35.69"], withIfNoneMatch="*"andIfMatch=<etag>, either failing with412 PreconditionFailed. A remoteVersionis the ETag.- The manifest is widened by compare-and-swap, and a writer whose swap fails re-reads and checks again.
- Remote access is configured by the AWS environment alone. DuckDB gets
CREATE SECRET (TYPE s3, PROVIDER credential_chain, ...), plusENDPOINT,URL_STYLE 'path'andCHAIN 'env;config'whenAWS_ENDPOINT_URLis set, so no instance-metadata probe hangs.httpfsand boto3 load on the first://URI, so a local workspace never runs the probe. ws.conis a cursor per thread, from athreading.localsubclass. Only a relation has a cursor of its own. Every other object looks upworkspace.conper call. A relation, a frame and aWorkingRecordbelong to the thread that built them.- A frozen node's folded axes and groups become shared tables,
dims_{dim}_{id}andgroup_{name}_{id}, read throughcast_declared.{id}is the revision, the base of aWorkingRecord, oruuid5of a standalone directory's URI.Record.dimsbuilds each frame per access. When two misses race tocreateone table, the one that fails reads the table the other created. as_relationrefuses a relation from another connection or cursor with aValueErrorthat states the fix.Revisionserialises as{"id", "parent", "store"}and validates back throughWorkspace.process().store(uri).revision(id), the one default left.Store.__reduce__does the same. Only deserialisation calls it implicitly, and itsdatabasedoes not travel. Code whose records must join deserialised ones callsWorkspace.process()by name.- A database file is a workspace, never the record. A location is not reused, nor a standalone directory's URI, because datarecord keys caches by id and assumes that an id's content never changes.
Recordmoves tolayered/record.py, beside its backingResolver, and nothing imports back.datarecord/record.pystill containsRecordLike,LayerDataand their value types, andlayered/revision.pystill containsRevisionandancestry.Tool.to_datarecord(model, con)andTool.results(model, con)requirecon, not a workspace or store, andon_connectiongoes.build(record)readsrecord.con.- Deleted:
DEFAULT_BASE_URI,DATARECORD_BASE_URI,_BASE_URIS,base_uri_of,default_connection,connect, thelayer_dirmacro, everybase_uri=None,read_json,Revision.create,Revision.get,Record.at,Record.over, publicwrite_record,commit,NewChild,Directory,TargetandResolver._cached_dims.write_record,layer_dir,connect,NewChildandDirectoryleave__all__, andWorkspaceandStorejoin it. A function that plans relations, such asfold_axis, keeps itscon.
Rationale¶
A required location replaces base_uri=None, which silently meant the process default.
child sits on the parent, because a parent is always its own store's. One call per destination fits destinations that need and return different things.
commit did three jobs, and the caller now does each: picks w.edits or w, names the parent, and calls rollback(). datarecord does not refuse rev.child(w), which writes the whole record as a correct but larger layer.
A revision file per UUID is write-once like a layer, so two processes never write one object. One listing per thousand revisions replaces a read per revision. The revisions cache cannot go stale, because a parent never changes.
boto3 is what Manta already runs, at the cost of its size and of S3 alone. Its conditional writes apply to the revision files and the manifest, but not to the layer, whose parquet DuckDB writes. A fresh UUID and an unreachable failed write cover that.
The guarantees hold only as far as a server honours the headers:
| SeaweedFS | If-None-Match and If-Match on PutObject |
|---|---|
| before 3.97 | ignored |
| 3.97 to 4.17 | checked before upload, so two racing writers can both pass (#7154) |
| 4.18 and later | checked under the write lock, so exactly one of two racing writers succeeds (#8802) |
On Manta's 4.05 the revision files are safe in practice, but the manifest's compare-and-swap can lose a widening, so Manta's pin moves to 4.18 or later. AWS S3 has honoured If-None-Match since August 2024 and If-Match since November 2024.
The tree cannot live only in a DuckDB file, which takes one writer and cannot be remote. A DuckDB file is still useful: staged edits survive a crash, the revisions cache persists, and DuckDB can spill.
Implementation notes¶
WorkspaceandStore, the localFiles, the tree, cursors, shared fold tables and the deleted globals, in one commit. These parts depend on each other, so a commit with only some of them would leave datarecord broken.- Writing:
root,child,ws.write, andedits. - Remote
Filesand conditional manifest writes. - Tools take
con.
On DuckDB 1.5.5, eight threads on their own cursors all succeed. So do sixteen threads taking a first cursor at once, fifty times over.
The remote tests run against SeaweedFS 4.18 or later in a container, because a mock would ignore the headers. New tests check a deserialised revision's ancestry, concurrent reads and cache misses across threads, create-only revision files, two widenings of one manifest, an unlisted revision after a failed remote write, a refused non-empty ws.write, a fold table read under a widened schema, and child(w.edits) on the wrong revision.
The build departed in the remote Files. boto3 uses path-style addressing whenever AWS_ENDPOINT_URL is set. A failure setting up httpfs raises. A 403 on a read or exists counts as a miss, and a 409 counts as a refused write beside 412.
An older location has no revisions/ and reads as an empty tree, fixed by exporting the revisions table as one JSON file per row.
Rejected ideas¶
- One object for location and connection: two locations could not join.
store.write(source, parent=...)repeats the store,rev.writereads as changingrev, and a target union needs overloads.- Revision files inside a layer: a layer directory stays what the layer wrote, and the tree lists in one listing.
- One
revisions.parquetor a shared DuckDB file: not write-once on S3. gen_random_uuid(): the layer directory needs the id before any row exists.fsspecors3fshas no create-only write, and obstore is a second S3 client.- A
remote=override: it would not apply toWorkspace.process()in a deserialising task. - A secret from
KEY_IDandSECRET: loses tokens and roles. - A dict keyed by
threading.get_ident(), whose ids are reused, or aContextVar, which a task runner copies into workers. - A thread check: it misses calls a frame makes on its own cursor.
- Re-issuing a relation on another cursor: impossible over a Python object.
- Copying a foreign relation: an unasked-for full materialisation.
Open issues¶
- Sweeping orphaned layers of a failed remote write.
ws.new(schema), aWorkingRecordwith no base.- A handle to resume staged edits from a database file.
- A
createdfield on a revision, and reading overhttps://.