Skip to content

datarecord

datarecord is a Python package that stores a model's input data, and every scenario built on it, as one versioned dataset. It is for modellers who run a framework such as PyPSA over many variants of one dataset. With datarecord you keep every variant, see how each differs from its base, and read any of them back as one dataset.

A base record and two scenarios in one store, and one scenario read back A schema declares the dims entity, bus and snapshot, and the attributes p_nom over entity, p_max_pu over entity and snapshot, and marginal_cost over entity. A commit writes the base revision into one store of parquet directories, with p_nom 100 for wind1 and 50 for gas1. Two scenarios branch from base and hold only the rows they change: high wind sets p_nom for wind1 to 150, and carbon price sets marginal_cost for gas1 to 80. Reading high wind returns one record with p_nom 150 for wind1 from high wind and 50 for gas1 from base. The PyPSA tool, pandas or polars, and DuckDB SQL read that record. schema declares what a record may hold one store, parquet directories base p_nom carbon price marginal_cost only the rows it changes high wind p_nom only the rows it changes read high wind p_nom one record, deepest layer wins dim entity bus snapshot attribute over p_nom entity p_max_pu entity, snapshot marginal_cost entity entity value wind1 100 gas1 50 entity value gas1 80 entity value wind1 150 entity value wind1 150 gas1 50 commit branch branch read PyPSA tool pandas / polars DuckDB SQL

A schema shapes the base record, two scenarios hold only the rows they change, and reading high wind returns one whole record.

A modelling study often ends with dozens of variants of one dataset. One adds wind capacity, another sets a carbon price, a third swaps the demand series. Kept as full copies, the variants drift apart, and you cannot tell which rows differ without comparing every file. datarecord stores each variant as a layer that contains only the rows that differ from its parent, so a scenario costs the rows it changes. To see what a scenario changes, you read its layer.

What you get

You build a scenario by editing a parent revision (one saved version in the store) and committing the edits. store.commit writes the edits as a new layer under that parent. When you read the scenario, datarecord combines its layers from the base down. Where two layers set the same row, the value from the layer nearer the scenario is the one you read. If many reads start from one revision, you can materialise it, so that later reads do not combine the layers below it again.

Each record has a schema, which lists the dims (the labels that values vary along) and the properties of each label. It also lists the attributes, with their dtypes and units. datarecord checks every edit against the schema and refuses it before it writes anything. For example, if the schema declares p_nom over entity, datarecord refuses a value of p_nom per snapshot. An attribute that the schema does not list raises KeyError. So every record you read from disk has the shape that its schema declares.

On disk, each layer is a directory of parquet files (a common columnar file format). You can open these files with any parquet library, such as pandas or polars, without datarecord installed. However, to read a scenario combined across its layers, you need datarecord. In Python, every read returns a DuckDB relation (a query that has not run yet). DuckDB reads the files only when you call .df(), .pl() or .to_arrow_table(). The calls are the same whether the store is on a local disk or on S3.

datarecord does not depend on any modelling framework. It needs only duckdb, pydantic and mathspec. A framework such as PyPSA reads a record through the RecordLike protocol, and a workflow engine writes one. So neither needs to know how the other works. The PyPSA tool, which translates between a record and a PyPSA network, installs as an extra.

Example

This example branches a scenario high off a committed revision base, raises the capacity of wind1 from 100 to 150 and reads both back:

from datarecord import WorkingRecord

w = WorkingRecord(base.record)
w.set("p_nom", 150.0, entity="wind1")
high = store.commit(w, meta={"scenario": "high"})

high.record.attributes["p_nom"].df()
#   entity attribute  breakpoint  value
# 0  wind1     p_nom         NaN  150.0
# 1   gas1     p_nom         NaN   50.0

store.layer(high.id).attributes["p_nom"].df()
#   entity attribute  breakpoint  value
# 0  wind1     p_nom         NaN  150.0

high reads as the whole record, with wind1 at 150 and gas1 unchanged. Its layer on disk contains one row. base still reads wind1 at 100.

The scenario tutorial builds this from an empty store in about ten minutes.

Install

pip install "datarecord @ git+https://github.com/energy-models/datarecord"          # core
pip install "datarecord[pypsa] @ git+https://github.com/energy-models/datarecord"   # with the PyPSA tool
pip install "datarecord[remote] @ git+https://github.com/energy-models/datarecord"  # with records on S3

The package on PyPI, 0.1.0a1, predates the design these pages describe, so install from GitHub until the next release. The install brings mathspec with it, at the commit that pyproject.toml pins.

Installation lists other package managers.

Where to go

Build a first record, branch a scenario and solve a record with PyPSA, one step at a time.

Finish one task: declare a schema, read, select, edit, branch, store, or hand a record to PyPSA.

What datarecord accepts, returns and refuses, and the format on disk.

Why the design is the way it is, what it rejected, and what it leaves open.

To change datarecord itself, start at Development.