Changelog¶
All notable changes to datarecord are documented here. The format follows Keep a Changelog.
[Unreleased]¶
Added¶
select(record, **labels), exported besidesettle, andRecord.selectreturn aSelection: aRecordLikeholding any record at the labels named, as xarray'sseltakes them. One label squeezes its dim out of the schema, its attributes' rows settled along it by the broadcast rule; a list, a slice or a partial date string keeps it. A keyword naming a multi-dim's coordinate narrows its tuples, and one naming a property narrows its subject. A label whose property names a dropped label goes, and so does a tuple naming one, until nothing more goes.Selection.selectmerges its labels and selects the base again, andSelection.liftputs the squeezed labels back on rows computed over the view. Attributes stay plans on the base's connection.- Properties. The schema has a
propertiessection (PropertySpec) that declares, per dim or multi-dim, the values that describe one label. A record stores them as columns ofdims/<dim>.parquet. A caller names a property assubject.name, as inset("bus.x", ...)anddefault_bykeys such as{"entity.ctype": {...}}.Schema.resolve(name)returns the spec of a name, andSchema.column_typetakes the property's subject. A property may share its name with an attribute, a dim or a property of another subject. - A property declares exactly one of
dtypeandone_of: <dim>. A property withone_ofrefers to that dim, and its values are live labels of that dim.setandaddcheck them, and the writer checks every layer.remove(dim, labels)fails while such a property still holds a label. - A property that refers to a dim with
one_of, on a subject inpartial, a multi-dim among them, categorizes the subject. Categories select labels inflags(ctype="Generator")and inset("p_max_pu", 0.9, ctype="Generator").default_bygives a default per category.Schema.categorizinglists the categorizing properties of a subject. - A dim of dtype
strmay declareone_of: [...], as in mathspec. Its column is anENUMof those labels, and its axis is the list, in order. WorkingRecordedits any dim:add(dim, labels, **properties),remove(dim, labels)andset(name, value, **coords).setwrites attributes and properties, and a property keyword selects labels whensetis called.- A multi-dim is a broadcast coordinate like any dim. A NULL multi-dim value
means every tuple of the multi-dim, and
partialmay name a multi-dim. Workspaceholds the DuckDB database andStoreholds one location. Each layer underlayers/<id>/carries its ownmanifest.json, aManifestholding its id, its parent, the full schema it was written under, the files it wrote and a flatmetaof strings, integers, finite floats and booleans. A read opens exactly the files the manifest lists and globs nothing. ARevisionis its location and id: it compares, hashes and serialises as them, and reads its ancestry in a new process.rev.manifestandrev.metaanswer from the workspace'srevisionsandschemastables, which cache each manifest whole.ws.conis the cursor of the calling thread. The write and edit boundary rejects a relation from another connection or thread.Workspace(database, con=..., name=...)attaches its database asname,datarecordby default, on a connection of its own or on a caller'scon, such as a server's, whose default database it leaves alone. It creates its tables there, so a user's file opened as a workspace gets datarecord's tables beside its own.ws.close()closes its cursors and detaches the database, and closes the connection only where it opened it. Its S3 secret is nameddatarecord_s3, so it replaces no secret of the caller's.- The prefix
_dr_is reserved: a table or view whose name starts with it, in the default database or temp schema of a workspace's connection, may stand in for the data datarecord's SQL reads. - Writing is
store.commit(source, parent=..., meta=...), withparent=Nonefor a new root, andws.write(source, uri, meta=...).store.claim(parent=, meta=)hands out a layer id by a create-onlyclaim.jsonbefore any data is written, andclaim.commit(source, meta=...)writes the data intolayers/<id>/and then the manifest, create-only. Until then the layer is pending and no read reaches it. A commit may addmetakeys but not change a claimed one. A child whose schema is notcompatible_withits parent's, or a parent from another store, is refused.store.commit(w)writes the edits of aWorkingRecordas a child of the revision it was built over, and refuses one built over anything else, namingws.write(w, uri), which writes the flattened record. Each claim and manifest is written once, create-only. - A
Claimserialises and pickles as its store's location and id, and reads its parent,metaand claim time back from itsclaim.json, so a job in another process commits it.store.claims()lists the pending layers, oldest first, with theirmeta, andstore.revisions(meta={...})returns the revisions whosemetaholds every pair given. store.cleanup(ids=[...])andstore.cleanup(older_than=timedelta(...))delete pending layers, and refuse an id that is not pending. Nothing else deletes.- A store or standalone directory on
s3://, through boto3 behind thedatarecord[remote]extra and configured by the AWS environment. Claims and manifests are written create-only, and each writer checks its schema against its parent's manifest, which never changes, so two writers never race. Tool.to_datarecord(model, con)takes the connection that its frames are planned on.- Removing an entity removes the group tuples that name it, and the values of
a tuple go with it. An
addof the name revives the entity alone. A layer that adds a tuple naming a removed label, or removes a label that a tuple names, fails at write. - A layer that changes an axis or a group restates it whole, in order. A label
is removed where a restatement omits a label that the previous one held,
and its order is the file's row order. The deepest file for an axis or group
is the answer for a node. A
WorkingRecordstages an edited axis or group whole, copied from the base at its first edit. A materialised node names the layer holding each axis's deepest file inresolved/dim_owners.parquet. WorkingRecord.setchecks that an attribute value names a label that its dim's axis, as staged, holds, for every dim, and a tuple that its group, as staged, holds.addchecks that each label of a tuple is on its axis as staged, and that a value of a property withone_ofis a label of that dim as staged.write_layerchecks every attribute row against the final axes and groups of the layer, and every group tuple against the final axes. Each check is one DuckDB anti-join.- A schema is checked at load. A dim, group, group coordinate or attribute
name that is not a plain identifier (mathspec's rule: ASCII letters, digits
and
_, not starting with a digit), starts with_, is a column that datarecord writes itself (value,layer_uuidand others, ignoring case), or differs from another name only in case fails with an error that names each offender. Based on #26 by @FBumann. - A manifest with an unknown key fails to load with pydantic's
ValidationError. - The PyPSA tool declares buses and carriers as dims. A bus is a label of
busand a carrier a label ofcarrier. The properties ofbusarecarrier,type,x,y,unitandlocation, and those ofcarrierarecolorandnice_name.v_nom,v_mag_pu_*,co2_emissions,max_growthandmax_relative_growthare attributes over their dim.marginal_price,v_mag_puandv_angare results overbus, andbuildsets them back.to_datarecordadds every carrier that a component names andcarrierdoes not list, and stores""as NULL.verifyreports an entity attached to no bus. - The PyPSA tool declares the dims
ctypeandrolewithone_of, andentity.ctype,entity.carrierandconnection.roleas properties.partialis{"entity", "bus", "scenario", "connection"}. - PyPSA SubNetworks are labels of the dim
sub_network, without a suffix.sub_network.carrierrefers tocarrier, andslack_busis a property oversub_network. A solve that has SubNetworks restates thesub_networkaxis, and a solve without them restates it empty where the record has SubNetworks.SubNetwork.objis not written, andbuildgives each SubNetwork a fresh one, so a built network passes PyPSA's topology code and optimises. - A static column that PyPSA's registry lacks is a property of its type's
subject if it holds text, such as pypsa-eur's
Bus.countryasbus.country, or if as an attribute its name would hold two shapes, such as a customGenerator.v_nom. Any other, such asBus.nom_max_onwind_2050, is an attribute over its type's dim.buildsets either back on the types that carried it.to_datarecordrefuses a custom property that types of one dim carry with different dtypes.Requirements.columnsis gone, as no custom column clashes any more. - The PyPSA tool stores per-port results on connections like per-port inputs.
p0,p1,p2and a one-port type'spare one resultpover theconnectiongroup, with a row per(entity, bus)that keeps the port's sign.qworks the same way.Link.pandProcess.parep_internal, and a three-port Link'sp2is a declared result. - The PyPSA tool stores a Process's
rate0,rate1,rate2as one inputrateoverconnection, static or per snapshot, andbuildputs each row back in its port's column.ratedeclares no default, so its ports default to-1and1. - The PyPSA tool round-trips multi-period networks.
snapshotis a multi-dim over(period, timestep), PyPSA's own index, so a snapshot keeps its tuple whether or not a timestep repeats across periods, and a single-period network keeps asnapshotdim of timestamps.buildrestores snapshots, investment periods, time series and both weightings. - The PyPSA tool round-trips snapshot and period weightings:
snapshot_weighting_objective,_storesand_generatorsoversnapshot,period_weighting_objectiveand_yearsoverperiod, andscenario_weight. - The PyPSA tool round-trips a network's shapes. A shape is the property
shape_{type}of the label it outlines, its geometry as WKT: aBusshape isbus.shape_onshore, and acomponentthat is no PyPSA type, such as pypsa-eur'scountry, is a dim of its own whose labels are the shapes'idx. Such a dim is no dim of the record's own, and a network is built from no selection squeezing it. A custom text column named like a dim refers to it, so pypsa-eur'sbus.countryis{one_of: country}. The CRS iscrsin the schema'smeta, andbuildrestoresn.shapesin that CRS.to_datarecordrefuses a shape without acomponent,idxortype, two sharing(component, idx, type), one naming no member of its PyPSA type, a custom column namedshape_...or like a key dim (scenario,entity, ...) and a custom value naming no label of the dim it refers to.buildnames each shape"{component} {idx} {type}". PyPSA.to_datarecord(n, con, results_only=True, base=record)writes a solve as a thin layer. The layer restates the SubNetworks ofbaseand sets NULL on the results ofbasethat it does not write again. A NULL series value on export leaves the component at its static value.PyPSA.buildtakes anyRecordLike, soPyPSA.build(PyPSA.to_datarecord(n, con))builds the network its committed record does.Record.register(name)creates DuckDB views of a record's dims, properties and attributes in the schemasname.dims,name.propertiesandname.attributesof the workspace's database, asdatarecord.name.dims.bus. Over aWorkingRecordthey are live and show every edit.Workspace.unregister(name)drops them, andWorkspace.registered()lists them.WorkingRecord.close()drops the record's staging tables, as does garbage collection once no relation taken from it lives, andwith WorkingRecord(base) as w:closes it on exit. A closed record raisesValueErroron every read and edit. A registration of the record keeps the tables untilWorkspace.unregister(name). The workspace keeps its tables out of its database'smain: a revision's fold in_fold."<id>"and aWorkingRecord's staging tables in_staged."<id>", soregisterrefuses these two names. A call's intermediate tables are temporary tables of its cursor. A workspace reopened on a file canunregisterthe registrations it lists.Record.close()releases the record's folds, andwith record:closes it on exit. The workspace drops a revision's fold once no open record or registration holds it. Each relation a record hands out keeps the record, and its tables, alive, as does a relation derived from it by its methods, so a chained read stays valid; a caller's own SQL over one does not. A closed record raisesValueErroron every read, andRevision.recordbuilds a new one once its cached record is closed.Workspace.registered()lists the schemas each registrationheld, which a reopened workspace holds for it again.PyPSACollection, a tool wrappingPyPSA, writes and builds apypsa.NetworkCollection, each member throughPyPSA, as a full export or as results overbase. The collection's dim is the name of its one index level. Overperiodit is a myopic solve, written as the one network holding every member's period, whose static values must agree among the members holding the entity active, andbuildgives one multi-period network per period,record.select(period=[p]). An asset a later member lacks retires there, itslifetimeending at the first period lacking it, and one held again after a gap is refused. Over any other name, a dim of the record's own, that dim is added to the schema and topartial, an attribute whose values differ across the members varies over it, and one whose values agree is written once, as declared;buildgives one network per label,record.select(dim=label). A dim the PyPSA schema declares,scenarioincluded, is refused: a stochastic network is written byPyPSA. Members hold one set of snapshots, components and custom columns per type, their axes are one union in one order, checked with their properties, which must agree, before any attribute, and every check runs whento_datarecordis called.buildtakes the record's dim of its own, elseperiod, andverifyreports what any member, narrowed on its own, lacks, and each dim of its own of a record declaring more than one inRequirements.unsupported_dims, a new field for dims a tool has no place for;Requirementsmerge with|. A record has at most one dim of its own: a collection over another is refused over abasethat has one.PyPSAbuilds no record with a dim of its own,verifyreporting it inunsupported_dims, and refuses to write results over one.- A collection's results over a
baseholding labels of its dim the collection lacks write every result at each member's label and leave the dim's axis and the SubNetworks tobase. Over abaseholding the dim, each member restates its series at its own label alone. A network built from the selection at one label of the dim,record.select(weather_year=2013), writes its results over that selection as that member. PyPSA.buildandPyPSA.verifytake a selection,record.select(...), as anyRecordLike. One label ofperiod,scenarioor a dim of the record's own squeezes the dim: a single-period network indexed by its timesteps holding the entities active in that period, a global constraint for another period dropped, a deterministic network, or the member of a collection at that label. A list keeps the dim, renormalising the kept scenarios' weights, and a list of periods holds the entities active in at least one of them. On top of the selection, a period left without snapshots goes, and an entity attached to a dropped bus. A selection squeezing any other dim, such assnapshot,timesteporbus, or holding no snapshot, is refused. A network's results are written over the selection it was built from,base=view, and lifted back to the labels it squeezes.
Changed¶
- Breaking: the workspace's database is never a cursor's default, so SQL
names its tables and registered views in full,
datarecord.main.registeredordatarecord.edit.dims.bus, or runsUSE datarecordonce on its own connection and writesregisteredoredit.dims.bus;ws.registered()reads the table. - Breaking: the schema models (
Schema,Dimension,AttributeSpec,PropertySpec) are frozen; derive a changed one withmodel_copy(update=...).Schema.digestis the sha256 of its manifest JSON. - Breaking:
RecordLikedeclarescon, the connection its relations are planned on; a hand-writtenRecordLikeadds it. WorkingRecordover aRecordLikethat is noRecordraises aTypeErrornaming the fix,WorkingRecord(ws.write(source, uri)).-
Breaking: the PyPSA dialect declares every component attribute over
scenario, except those PyPSA keeps the same in every scenario (*_nom_extendable,*_nom_opt,committable,lifetime, ...), so a staticp_nomthat differs per scenario round-trips (#28). A keyedset("p_nom", {...})is then open inentityandscenario; pass the labels asentity=[...]or as a series indexed byentity. -
A workspace validates each distinct schema once. Revisions sharing a schema hand out one
Schemaobject, so reading a long lineage no longer re-runs the schema's checks per layer. ASchemaobtained from a revision is shared: copy it before changing it. - A materialised node no longer copies its axes into
resolved/dims/.resolved/dim_owners.parquetnames, per dim, the layer whosedims/<dim>.parquetis the deepest at or below the node, and a read opens that file. Itshas_deadcolumn says which dims have dead labels inresolved/dead/, so a read opens those by name and lists no directory.Store.dim_owners_uri(id)is where it lives. A node materialised before holds no such file and fails the read: delete itsresolved/and materialise it again. - datarecord requires DuckDB 2.0. A relation it returns is a DuckDB 2.0
relation: it re-runs on every fetch and has no
fetchone(). -
duckdbis capped at 2.0.0, the newest release the suite passed on. -
The PyPSA tool melts a time series one entity at a time, so a parquet row group holds few entity names and encodes them by dictionary. The
p_max_pufile of an 80,000-generator hourly network shrinks from 4.7 GB to 2.3 GB. -
Every parquet file datarecord writes is zstd-compressed parquet v2, with a row group's dictionary capped at 4 KB, so a series' snapshots are delta-encoded and its values byte-stream-split. That
p_max_pufile comes to 406 MB. Attribute files are written without keeping row order, so writing a long series no longer spills tens of GB; dims files keep theirs, which is the axis order. -
The
pypsaextra requires PyPSA 1.4. -
PyPSA.to_datarecordtakesbaseon a full export too, and drops theValueErrorforbasewithoutresults_only. Overbasea static value is left out only where it repeatsbase's value, else the default, so a value set back to PyPSA's default over a parent holding another is written and no longer lets the parent's show through. A series, a weighting among them, repeatingbase's at every snapshot is left out too, one differing anywhere written whole, and an input at a port or a custom column is compared withbase's, so the layer holds what changed rather than every series. -
Groups are multi-dims. A dim declared under
dimensionswithover, and neitherdtypenorone_of, is a multi-dim: its labels are tuples of other dims' labels, carried as one struct column named for it. Its table isdims/<multi-dim>.parquet, keyed by that column with its properties beside it;groups/andresolved/groups/are gone, so every record with a group is written again.record.dims["connection"]is the struct column alone andrecord.properties["connection"]addsrole.addandremovetake a multi-dim's tuples as a frame holding its struct column, soadd("connection", record.dims["connection"])round-trips, or one with a column per coordinate instead, or as a list of mappings from coordinate to label.connection.busnames a coordinate whereversubject.namestands, andsetrefuses it. No property refers to one. New onSchema:multi_dims,over,multi_dims_of,check_multi_dim_columnsandcoordinate_over;dimslists every dim but the multi-dims. The PyPSA tool declaresconnectionunderdimensions. - A dim with
one_ofhas its axis derived from the list:record.dims["ctype"]is the list, and nodims/<dim>.parquetis written or read for it, nor is it named inresolved/dim_owners.parquet. The writer refuses aLayerDatahanding one over, a read refuses a layer holding one, and aRecordLike's axis of it is dropped where it is the list, same labels in the same order, and refused otherwise, naming a NULL, a repeated, missing or extra label, or else the order.addandremoveon it are refused, and a property declared on it is refused at load. A record carrying such a file is written again. New onSchema:one_of_dims;Schema.require_listedisrequire_one_of. The PyPSA tool no longer writes thectypeandroleaxes. - A multi-dim in
partialis categorized by a property that refers to a dim withone_of, as a dim is:connection.rolecategorizesconnection, so adefault_bykeyedconnection.roleloads,Schema.categorizing("connection")listsrole, andflags(role="output"),default_for(name, role="output")and thesetkeywordrole="output"select by it. A role keyword picks the tuples of an attribute overconnection, narrowed by a coordinate keyword, and of a property of it:set("connection.note", v, role="output")patches the output connections, asset("entity.note", v, ctype="Generator")does the generators, andctype="Link"on it the connections of links. - The design pages are grouped by topic: the data model, layers, using a record, storage, frameworks and the package. Each old design page URL redirects to its new page. The usage pages are organised by task, and the API reference uses the design section names.
- An attribute over a multi-dim carries it as one struct column named for it,
with a field per coordinate, such as
connection STRUCT(entity, bus). Its table is keyed by that column, anddims/connection.parquetisconnection STRUCT(entity, bus) | role. The PyPSA tool declares nobusattribute. A port's bus is the connection tuple. - The dims that a multi-dim draws on broadcast. A per-bus value with
bus = NULLholds for every bus, andbusmust be inpartialto patch one bus at a time.WorkingRecord.setstill takesbus=. Naming a multi-dim next to a dim that one of its coordinates is named for, as indims={"connection", "bus"}, is an error. entityis an ordinary dim. It broadcasts, and it must be inpartialto be patched or categorized per label.entity_typeisentity.ctype.- An attribute reaches every label that its
dimsaddress, whatever the categories of the labels. A NULL on a categorized dim reaches every label.flags(**categories)holds every attribute with a row that reaches a matching label. The PyPSA tool keeps the per-type vocabulary.buildleaves out a value that its type does not take and reports it inRequirements.ignored, andto_datarecordrecords custom entity columns per type in the schema'smeta. - Every layer holds its attributes in one
attributes/directory, and a result inherits through layers like any attribute. The PyPSA tool declares its outputs as attributes, andbuildcopies them. record.dims,propertiesandattributesreturnDuckDBPyRelations.FramesandLazyFramesare nowRelationsandLazyRelations.set,addand the writer take a relation or a pandas, polars or arrow frame, which DuckDB reads with its own scanner. They refuse timestamps with a time zone and cast a column only to a wider type of its kind. A dtype is mathspec's:float,int,bool,strordatetime.- The owner map's key is the
partialdims and groups plusattribute. A dim or group outsidepartialis owned whole, so a layer that patches one connection's value restates every connection's. The PyPSA tool declarespartialso that it keeps per-connection ownership.Schema.input_keyisSchema.owner_map_key,Schema.input_columnsisSchema.owner_map_columns, andRequirements.unsupported_keysreports"owner_map_key". - A
Recordis constructed fromrevision_id,sourcesandworkspace, and reads under its last source's schema. A caller reads it through its mappings:rev.record.sources,dims[dim],properties[dim]andattributes[name].attributes[name]raisesKeyErrorfor an undeclared attribute. PyPSA.buildraisesTypeErrorfor aRecordLikethat is not aRecord.Schema.long_columns_forandSchema.coordinates_ofraiseKeyErrorfor an attribute that the schema does not declare.LayerDatadeclares nofrozen, and itsaxes(),groups()andattributes()returnset[str]. A wholeRecordreaches the writer through the adapter for aRecordLike.- The modules live in the subpackages
datarecord.schema,datarecord.resolveanddatarecord.store.datarecord.recordisdatarecord.protocols,datarecord.filesisdatarecord.store.files, anddatarecord.layered.categoriesisdatarecord.schema.categories. The dtype helpers moved fromdatarecord.ducktodatarecord.schema.dtypes. Every public name imports fromdatarecord. - A
WorkingRecordfolds its staged layer onto the folded axes, groups and removed labels of its base, which the workspace keeps as tables. Folding the staged edits opens no file of the base. - A label or group tuple that one layer removes and a later layer adds again takes its place where it is added again, in a working record, a written record and a materialised node.
- A key held twice in a
dims/orgroups/file, and a column that is neither the key nor a declared property of its subject, fail at read.write_layerrefuses a restatement that orders the labels of its parent differently. - A reference to a removed label in a group tuple or a property is checked in every subject that a layer restates, whole. A reference that an ancestor left dangling fails the next layer that restates its subject.
- A layer's id is its manifest's. A standalone directory is one layer with
no parent, and
ws.atrefuses a manifest naming one, a layer of a tree being read alone throughstore.layer(id). - A value over a dim or group outside
partialthat names a removed label is dropped at read, including the per-bus values of a removedbuslabel. A materialised node keeps the removed labels inresolved/dead/. attributes[name]returns the owned rows as written. A NULL coordinate stays NULL, rows of two layers may cover one coordinate, and the row order is unspecified. A consumer settles each coordinate by the broadcast rule, the row naming every dim the others there name winning. The owner map holds each layer's keys as written, plus one row naming both wherever a deeper layer's row crosses a parent's, so it grows with the rows layers write rather than with the labels a broadcast reaches.- A deeper layer owns only the coordinates its rows reach. A shallower row keeps every coordinate that no more specific row covers.
- A label removed and added again takes every broadcast covering it, written before or after the removal, and no row that named it below the removal.
flags()counts a broadcast that more specific rows shadow at every label, and reports invariesandbroadcastonly the dims an attribute uses.WorkingRecord.setwrites a row that leaves a dim outsidepartialNULL out along the labels a parent row names there. The fold refuses a layer written otherwise that leaves such a crossing, naming the layer, the attribute, the dim and the key.- The writer refuses a
dims/file of a dim that the schema does not declare, as it refuses an undeclared attribute. A descendant declaring the dim would read its labels. - The writer accepts two rows that each name a dim the other leaves NULL
where a row naming both is there, so
ws.write(w, uri)writes a record of several layers. Asetstaging two rows that cross with no row naming both is refused. - The fold reads each layer under the schema in its own manifest. Where a
deeper layer's schema is wider, the map folded so far, materialised or not,
is extended to it, so a materialised node keeps counting after a descendant
adds a dim to
partial, and a workspace holds one fold table per node,owner_map_<id>. A layer that restates a key along a dim outsidepartialand omits a label its parent held there deletes that value, and a later widening ofpartialno longer brings it back. - The fold of a layer onto the map below it is one query whose joins its
schema fixes, whatever named sets its rows hold (
fold_layer, replacingfold_attributes). A frozen layer leaving a dim outsidepartialunfilled is refused when a workspace first folds it, not on every read. - Every
Recordlists every dim, property subject and attribute its schema declares, a typed empty relation where no layer holds a row, rather than raisingKeyError. - A relation of a
WorkingRecordis planned without reading a staged row, only the frozen layers' rows, so one built before an edit reads it. Every staging table exists from the start, an axis holding a copy of the base's, androllback()empties them rather than dropping them. Each edit runs in one transaction, which also refreshes the staged owner-map rows the fold,flags()and the write read. - A
setwhose row crosses a staged row, with no staged row naming both, also stages the row naming both with its value: the later edit wins where the two meet, rather than the read failing. - The edits of a
WorkingRecordrestate an axis only where it differs from the base's, and write no attribute without rows. - A fold starts from the deepest proper ancestor whose fold is materialised or already held by the workspace, so a record opened on a child after a commit folds only the child's layer.
- A
Recordbuilds the coords of its frozen layers once per thread rather than on every relation it reads, about halving a warm read of a deep record. A write checks a new layer's axes against its parent's fold. - The cold fold of a deep record's owner map takes about a third of the
time: each layer's owner rows, the parent rows they leave and the pairs
the check reads are each built once, the crossing pairs are taken from the
check's, unions bind once per arm, the parts of the plan a schema fixes are
built once per schema, and a layer's flags test only the attributes it
holds.
Revision.materialisewrites the fold its workspace already holds rather than folding again. - A write builds each attribute and axis carrying a multi-dim once. It checks for a tuple with some fields NULL on the file it wrote, not on the source.
- An edit inside a transaction the caller holds open on the same cursor is
refused with a
TransactionExceptionthat says so, since an edit runs in its own transaction. Attr.resolveandSchema.resolvetakelabelsandflags, and settle each source by the broadcast rule overlabelsbefore a rename or a computation.settle(rel, labels, flags), exported fromdatarecord, does the same for one relation, dropping rows that name a label outsidelabels.- A failed write, local or remote, leaves its files in the pending layer rather than staging them locally and renaming the directory into place.
Removed¶
- Breaking:
PinnedRecordis gone. A write reads aWorkingRecordthrough its per-attribute plans directly. Group, the schema'sgroupssection,Schema.groups,Schema.group_columns,Schema.group_type,Schema.groups_of,Schema.group_key,Schema.check_group_columns,Schema.field_over,RecordLike.groups,LayerData.groups()andLayerData.group(),WorkingRecord.add_groupandWorkingRecord.remove_group.settakes no expression. Compute the new values fromattributes[name]andsetthe frame.Dimension.withinis gone. Every axis is keyed by its label alone, with labels unique across the dim. A nested dim is a property withone_ofon the parent dim, such assnapshot.periodwithone_of: period.addtakes no parent keys, andSchema.axis_keyis gone.- Traits are gone:
Trait,Schema.traits,Schema.attributes_forandSchema.properties_for. Schema.results,Record.outputs,Kind,set(..., kind=...)andTool.resultsare gone.- narwhals is not a dependency.
FramesandLazyFramesare replaced as listed under Changed. ResolverandRevision.resolverare gone, withwith_source,inputsand the enumeratorsaxes(),groups()andattributes()of the resolver.write_record,Revision.create, the source-lessRevision.child(),NewChild,Directory,commit,connect,layer_dir,DATARECORD_BASE_URI, the process default connection andon_connectionare gone.Record.attakes a workspace.Schema.membership_keys,Schema.group_coordinates,Group.coordinates,WorkingRecord.set'sindexed_by,entity_typesand theentity_typedim with its member files are gone.store.root,rev.child,Schema.id, the store's onemanifest.jsonwith its compare-and-swap, and therevisions/tree are gone, as is the local staging directory of a write. A store written before has no layer manifests, so it is unreadable and is written again.
Fixed¶
PyPSA.buildrefuses a record whose rows of an attribute name several labels of a dim PyPSA's registry declares it without, such as a generator'smarginal_costoverperiodin a multi-period record, rather than keeping one label's value, andPyPSA.verifyreports the attribute inRequirements.unsupported_values. The view at one label,record.select(period=[2030]), builds, asPyPSACollectionbuilds each period.Requirements.describelists the field as unsupported values.PyPSA.buildsettles an attribute's rows over every dim butsnapshotthat the record holds one label of, not only overentity,scenarioandconnection. Inrecord.select(period=[2030]), a generator with a row at 2030 and one leavingperiodNULL holds the 2030 value once, rather than both.- Reading a property's default keeps the rows in their order. A relation
filtered by several
IN (SELECT …)subqueries no longer comes back reordered. - The PyPSA tool keeps a network's own line and transformer types. A
LineTypeorTransformerTypethat PyPSA's standard library does not name is a label of the dimline_typeortransformer_type, every column of it a property, andbuildadds it to the library. A PyPSA-Earth network, whose lines use its ownZM_330kV_bison_2bundle, no longer builds back with lines that reference a missing type. As in PyPSA's netCDF export, a library row the network has changed is not written. - A results layer writes a series at the network's snapshots alone,
restating
base's values at the others, and over a multi-periodbasea result over no snapshot, such asp_nom_opt, for the entities active in the network's periods alone, sobase's results stand at every period, snapshot and entity the network does not hold. Writing one period's solve after another's no longer zeroes the earlier period's dispatch and capacities, and the write no longer buildsbasefirst. The network may carry any ofbase's investment periods, or none where it is written over the selection at one period it was built from; one carrying fewer than all is written withresults_only=True, and one holding a snapshot, bus or entitybaselacks is refused at the call. - The PyPSA tool declares a custom column of dtype
objectby its non-null values, so a column of bools, ints or floats holding a NaN, numpy scalars among them, is written asbool,intorfloatrather than failing asstr, and one of mixed types is written as text. - A results layer restates as NULL only
base's results at the scenarios, buses, entities and connections the network holds, so a re-solve of a selection keeps every other label's results. A result over noscenario, such asp_nom_opt, is one value across every scenario, so a re-solve of fewer scenarios than the record holds is refused rather than written for every scenario beside the others' joint solve. A re-solve of fewer buses or entities thanbaseleavesbase's SubNetworks, since it numbers its own over the selection alone. The results of a deterministic network over abaseholding scenarios are refused rather than written over every scenario. PyPSA.buildof a stochastic record indexesn.shapesby(scenario, name), as PyPSA indexes every other component.- A results layer writing a result equal in every scenario once, with a NULL
scenario, no longer restatesbase's rows of it at each scenario as NULL, which hid the new value at every scenario. PyPSA.to_datarecordrefuses a stochastic network whose scenarios hold different values of a property, such as a custom text column, rather than writing the first scenario's.- The writer's check that no coordinate of a file is held twice groups 64-bit hashes in passes of at most 100 million rows, so a file of hundreds of millions of rows is checked in bounded memory and spill.
- A pointwise
setleaving one of two dims outsidepartialNULL keeps the rest of the series. - A broadcast reaches every label the record holds, including one added below
a materialised revision or staged in a
WorkingRecord, so materialising changes no answer. setrefuses a frame column that the schema does not declare for the attribute, and names the column and the columns it takes.- A later
setreplaces the rows of an earlier one that it covers. A row is covered where, at each coordinate, it names the same label or leaves NULL, including labels that a property keyword or a group's fields select.set("p_nom", 0.9, entity=["a"])and thenset("p_nom", 0.95)reads 0.95 ata. The other order keeps both rows, andareads 0.9. - Within one layer the more specific row wins, so a layer that holds a default
and its exception, such as
set("p_nom", 0.95)and thenset("p_nom", 0.9, entity=["a"]), reads one value at each coordinate. A row that names a label of a dim is more specific than a row that leaves that dim NULL. Two rows that cover one coordinate, where each names a dim that the other leaves NULL, fail at write and at read, naming the attribute and the coordinate. - A layer with two rows of one attribute that name the same labels at one breakpoint fails at write and at read, naming the attribute and the coordinate.
- The writer casts every column to its declared dtype, as
setdoes. An attribute'svalueis stored as its dtype, and an integer column under afloatattribute isDOUBLE. A column of a type that does not fit its dtype, such as a string for adatetimedim or afloatproperty, fails with the attribute, property or dim and its dtype named. A column that holds only NULLs passes as the NULL of its declared type, insetandaddas well. The PyPSA tool's static rows keep an integersnapshot. - A label of a type that does not fit its dim's dtype fails with the dim and
its dtype named. This covers a series index, mapping keys, a keyword of
set, a multi-dim's coordinate keyword, the labels ofaddandremove, and a value of a property that refers to a dim.snapshot="2030-01-01"on adatetimedim is refused. addchecks that a property value is a label of its dim, and its error says to use one of the labels where the dim hasone_of.- Removing an entity drops the values of every group row that names it, in any
field. A
contractover(from, to)loses its values when either party is removed, andflags(ctype)counts such a row for the type of each party. materialise()writesowner_map/inputs.parquetlast and each file through a temporary name, so a node reads as materialised only when its cache is complete.- A map materialised under a parent's schema reads under a child's wider one.
- An attribute whose
dimsgained a dim reads its earlier rows, NULL along it, where that dim is inpartialtoo.flags()reports such rows as broadcast along the gained dim, whether it is inpartialor not, and whether or not the schema declared it before. - Reading an attribute over a
partialdim scales linearly with the length of that dim. Folding the owner map no longer compares every row with every label of the dim's axis, so the first read of a record with long series is linear. - A record whose layers are all written scans its owner map once for the layers that own each attribute.
- A pickled
Revisionunpickles throughWorkspace.process()and reads its ancestry, and pickles after.recordwas read. - A standalone record's
manifest.jsoncarries anidthatws.writemints, and the workspace keys its folds by that id. A record deleted and written again at one URI, or one relative path read from another working directory, reads its own folds.ws.atresolves a local path asws.storedoes and refuses a manifest without an id. ws.writeto an existing empty directory raisesFileExistsErrorup front and says that a standalone record is written to a URI that does not exist yet.PyPSA.to_datarecordmelts a time series whose component name holds a"or issnapshotorperiod, and accepts components whose names differ only by case, such asGenandgen.PyPSA.buildkeeps the time series of a component namedsnapshot, orperiodin a multi-period network. No name is pasted into theUNPIVOTorPIVOTas an SQL identifier.PyPSA.buildturns a per-connection input back into PyPSA's port columns.efficiencyon a Link's bus2 connection isefficiency2, static or per snapshot.- The PyPSA tool skips
Line.v_nomandShuntImpedance.v_nom, which PyPSA derives from the bus, and writes noBus.p,q,control,generatororsub_network, so a solved network exports.