Calliope

User-facing changes

|new| working SPORES mode, upgraded from v0.6 to include a selection of scoring algorithms (716).

|new| backend `set_objective` method, to switch between pre-defined objectives (716).

|changed| |backwards-incompatible| `from` and `to` parameters (to define start and end point of a transmission link) are now `link_from` and `link_to` (717).

|changed| |backwards-incompatible| `operate` and `spores` mode configuration options are now nested within the main configuration (e.g., `build.operate_window` is now `build.operate.window` and `solve.spores_number` is now `solve.spores.number`) (704).

|changed| coin-or-cbc is now available cross-platform on conda-forge and so is the recommended open-source solver to install a user environment with (744).

|changed| Upper bound pins for dependencies removed where possible, to minimise clashes when using calliope as a dependency in a project (744).

|changed| |backwards-incompatible| to ensure the model configuration always remains in sync with the results, `kwargs` in `model.build()` and `model.solve()` now directly affect `model.config` (704)

|changed| `template:` can now be used anywhere within YAML definition files, not just in the `nodes`, `techs` and `data_tables` sections (719).

|changed| Removed `inheritance` math helper function since we use `base_tech` for abstract base technologies and templates are now applied too early to be available later (719).
To refer to template inheritance, set a parameter within a template as all children will share the same value.

Internal changes

|changed| Moved to using `pydantic` to document and validate our configuration, rather than JSON schema (704, 717).

|changed| As with base dependencies, moved to pinning lower bound only for development dependencies (744).

|changed| Curtailed `calliope.AttrDict` relevance, with I/O moved to `calliope.io` and attribute access moved to `pydantic` models.
`AttrDict` is still used to merge overrides and templates into the model definition dictionary, before creating the `pydantic` models.

User-facing changes

|changed| "An overview of the Calliope terminology" information admonition to remove self-references and improve understandability.
Now also includes a visual depiction of how the different defined components connect together (699).

|fixed| Area-based parameters have appropriate documented units of `area` rather than `area^2` (701).

|fixed| Technology capacity lower bound constraints so that `[cap-type]_min` (e.g., `flow_cap_min`) is not always enforced if the `purchased_units` variable is active (643).

|changed| Single data entries defined in YAML indexed parameters will not be automatically broadcast along indexed dimensions.
To achieve the same functionality as in `<v0.7.dev4`, the user must set the new `init` configuration option `broadcast_param_data` to True (615).

|changed| Helper functions are now documented on their own page within the "Defining your own math" section of the documentation (698).

|new| `where(array, condition)` math helper function to apply a where array _inside_ an expression, to enable extending component dimensions on-the-fly, and applying filtering to different components within the expression (604, 679).

|new| Data tables can inherit options from `templates`, like `techs` and `nodes` (676).

|new| dimension renaming functionality when loading from a data source, using the `rename_dims` option (680).

|changed| cost expressions in math, to split out investment costs into the capital cost (`cost_investment`), annualised capital cost (`cost_investment_annualised`), fixed operation costs (`cost_operation_fixed`) and variable operation costs (`cost_operation_variable`, previously `cost_var`) (645).

|new| Math has been removed from `model.math`, and can now be accessed via `model.math.data` (639).

|new| (non-NaN) Default values and data types for parameters appear in math documentation (if they appear in the model definition schema) (677).

|changed| `data_sources` -> `data_tables` and `data_sources.source` -> `data_tables.data`.
This change has occurred to avoid confusion between data "sources" and model energy "sources" (673).

Internal changes

|changed| updated transmission technologies to/from -> link_to/link_from to avoid conflicts with protected `python` terminology.

|changed| Model configuration, data tables, techs/nodes data, math and general model definition now uses `pydantic`.

|changed| Model definition reading is now defined in a single place (preprocess/model_definition.py).

|changed| Moved YAML reading/importing functionality out of `AttrDict`. It is now part of our `io` functionality.

|fixed| Avoided gurobi 12.0 incompatibility with pyomo by setting the lower bound to v6.8.2.

User-facing changes

|fixed| Decision variable domain in math docs to use $\in$ instead of $\forall$ (652).

|fixed| Clarity of `flow_cap_min` description in documentation (653).

|changed| API/schema documentation is de-ranked in documentation search bar results (670).

|new| Math component cross-references in both directions ("uses" and "used in") in Markdown math documentation (643).

|fixed| Duplicated links in math documentation (651).

|changed| `node_groups` and `tech_groups` changed to a general top-level `templates` key,
accessed via the `template` key (replacing `inherit`) in `nodes` and `techs` (600).

|fixed| Contribution of `cost_om_annual_investment_fraction` to total investment costs, to not apply to depreciated costs (645).

|fixed| Math for multi-carrier variable export costs (663).

|new| Piecewise constraints added to the YAML math with its own unique syntax (107).
These constraints will be added to the optimisation problem using Special Ordered Sets of Type 2 (SOS2) variables.

|new| Direct interface to the Gurobi Python API using `!yaml config.build.backend: gurobi` or `!python model.build(backend="gurobi")`.
Tests show that using the gurobi solver via the Python API reduces peak memory consumption and runtime by at least 30% for the combined model build and solve steps.
This requires the `gurobipy` package which can be installed with `mamba`: `mamba install gurobi::gurobi`.

|fixed| Force a header row in tabular data loaded from CSV to (596).
Fixes issue where unexpected index levels can end up in the loaded data (573).

|fixed| Single element lists/sets in the model Dataset attribute dictionary are restored to lists/sets on loading from NetCDF (614).

|new| Decision variables and global expressions can have a `title` defined, which will be available in the model results as attributes of those components and can be used for e.g. visualisation (582).
Parameter titles from the model definition schema will also propagate to the model inputs.

|fixed| Backend parameter updates propagate correctly through global expressions in the order those expressions were defined (616).

|fixed| If setting `model.backend.verbose_strings()`, rebuilt model components from making backend parameter updates will automatically have verbose strings (623).

|fixed| Erroneous use of `dimensions:` in docs example of an indexed parameter (612).

|changed| `add_dimensions` to `add_dims` in `data_sources` definition to align with `dims` in indexed parameter definition (621).

|new| Allow extracting shadow prices into results by listing constraints in `config.solve.shadow_prices`, e.g. `config.solve.shadow_prices: ["system_balance"]` Shadow prices will be added as variables to the model results as `shadow_price_{constraintname}`, e.g. `shadow_price_system_balance`.

|new| Model stores key timestamps as attributes:

* `timestamp_model_creation`: at the start of `Model.__init__()`
* `timestamp_build_started`: at the start of `Model.build()`
* `timestamp_build_complete`: at the end of `Model.build()`
* `timestamp_solve_started`: at the start of `Model.solve()`
* `timestamp_solve_complete`: at the end of `Model.solve()`

Internal changes

|changed| `model._model_def_dict` has been removed.

|new| `CalliopeMath` is a new helper class to handle math additions, including separate methods for pre-defined math, user-defined math and validation checks.

|changed| `MathDocumentation` has been extracted from `Model`/`LatexBackend`, and now is a postprocessing module which can take models as input.

|new| `gurobipy` is a development dependency that will be added as an optional dependency to the conda-forge calliope feedstock recipe.

|changed| Added any new math dicts defined with `calliope.Model.backend.add_[...](...)` to the backend math dict registry stored in `calliope.Model.backend.inputs.attrs["math"]`.

|changed| Function vectorisation when creating backend component arrays uses `numpy.frompyfunc` instead of `xarray.apply_ufunc`, so that masking with a `where` array can be done at function calltime.
This requires pre-broadcasting all arrays being passed to the vectorised function, but reduces memory peaks in the Gurobi backend interface in particular.

|changed| Default parameter values are not used to fill NaNs when adding parameters to the backend, but when evaluating expressions.
This reduces memory footprint of parameter arrays.

|fixed| Removed unused debug parameter in `Model.__init__()`

|changed| Ruff linter checking was extended with pydocstrings, flake8-pytest, and pyupgrade.

|changed| Moved from black formatting to the Ruff formatter (black-based, but faster).

User-facing changes

|new| `mkdocs_tabbed` option when writing math documentation to file (`calliope.Model.math_documentation.write(...)`) which will add YAML snippets to all rendered math as a separate "tab" if writing to Markdown.
Requires the [PyMdown tabbed extension](https://facelessuser.github.io/pymdown-extensions/extensions/tabbed/) to render the tabs correctly in an [MkDocs](https://www.mkdocs.org/) project.

|new| List of pre-defined parameters given in the `pre-defined` math documentation, with references back to the constraints/variables/global expressions in which they are defined (either in the `expression` string or the `where` string).

|new| Units and default values for variables and global expressions added to the math documentation.

|new| Variables and global expressions can have a `default` value, which is used to fill missing array elements when doing math operations.
These default values ensure that `NaN` doesn't creep into the built optimisation problem math and are set to values that lead to them having no impact on the optimal solution.

|new| Utility function `calliope.util.schema.update_model_schema(...)` to add user-defined parameters to the model schema / update existing parameters using YAML schema syntax.
`calliope.util.schema.reset()` can be used to clean the model schema and return to the original, pre-defined schema.

|fixed| Timeseries clustering file can be a non-ISO standard date format.
Both the index and the values of the timeseries (both being date strings) should be in the user-defined `config.init.time_format`.

|fixed| the decision variable `purchased_units` is linked to `flow_cap` even if neither of the parameters `flow_cap_min` or `flow_cap_max` have been defined by the user.

|changed| `inbuilt` math -> `pre-defined` math and `custom` math -> `pre-defined` math in the documentation.

|changed| Calliope attribute dictionaries (AttrDicts) no longer sort dictionary keys on `union`. Key order is now: original dictionary key order + any new keys being added in the order they appear in the new dictionary.

|fixed| Dimensions with numeric data can be defined in tabular data _or_ YAML and will appear as numeric in the processed Calliope model input dataset.
If all dimension data can be coerced to a numeric data type (e.g. `["10", 100, "-1"]`), then it _will_ be coerced (e.g., `[10, 100, -1]`).

Internal changes

|new| `py.typed` file so that mypy recognises Calliope as a typed library when it is imported as a dependency.

|fixed| Spelling of Black config option `skip-magic-trailing-comma`.

v0.7 includes a major change to how Calliope internally operates.
Along with this, there are multiple changes to how Calliope models are defined and configured.
This requires adapting models to work with 0.7.
We group changes into those that are primarily user-facing and relevant for all Calliope users, and those that are primarily internal, and relevant only for Calliope developers.

User-facing changes

This section gives a brief summary of changes.
For more detail, see our migrating from v0.6 to v0.7 section in our [documentation](https://calliope.readthedocs.io/en/latest/migrating/).

|new| Storage buffers available in all technology base classes.

|new| Multiple carriers and different carriers in/out available in all technology base classes.

|new| `node_groups` added to match `tech_groups`.

|new| technology efficiencies split into inflow and outflow efficiencies.

|new| Technology capacities and efficiencies can be differentiated between technology carriers.

|new| Parameters can be defined outside the scope of `nodes` and `techs` using the top-level `parameters` key in YAML.

|new| Any parameters can be indexed over arbitrary dimensions, both core Calliope dimensions and new, user-defined dimensions.

|new| Non-timeseries data can be loaded from CSV files or in-memory Pandas DataFrames using the top-level `data_sources` key in YAML.

|new| User-defined mathematical formulations using the new Calliope math syntax can be loaded when creating a model.

|new| Shadow prices obtained from a dual LP problem can be read by using `model.backend.shadow_prices.get("constraint_name")`.

|changed| |backwards-incompatible| Updated to support Python versions >= 3.10.

|changed| |backwards-incompatible| Updated to Pandas >= v2.1, Pyomo >= v6.4, Xarray >= v2023.10.

|changed| |backwards-incompatible| Flat technology definitions, removing the distinction between `essentials`, `constraints` and `costs`.

|changed| |backwards-incompatible| Timeseries data is defined under the `data_sources` top-level key, not with `file=`/`df=` at the technology level.

|changed| |backwards-incompatible| Demand and carrier consumption values are strictly positive instead of strictly negative.

|changed| |backwards-incompatible| `model.run()` method → two-stage `model.build()` + `model.solve()` methods.

|changed| |backwards-incompatible| `model` and `run` top-level keys → `config.init`/`.build`/`.solve`.

|changed| |backwards-incompatible| `locations` top-level key and `loc` data dimensions → `nodes`.

|changed| |backwards-incompatible| `parent` technology parameter → `base_tech` + `inherit`.

|changed| |backwards-incompatible| Cost parameters are flattened and use the indexed parameter syntax.

|changed| |backwards-incompatible| `links` top-level key → transmission links defined in `techs`.

|changed| |backwards-incompatible| Various parameters/decision variables renamed (namely `energy_` → `flow_`, `carrier_prod`/`_con` → `flow_out`/`_in`, and `resource` → `source_use`/`sink_use`).

|changed| |backwards-incompatible| Various configuration options renamed.

|changed| |backwards-incompatible| `force_resource` technology parameter → `source_use_equals` / `sink_use_equals`.

|changed| |backwards-incompatible| `units` + `purchased` decision variables → `purchased_units`.

|changed| |backwards-incompatible| Parameters added to explicitly trigger MILP and storage decision variables/constraints.

|changed| |backwards-incompatible| Structure of input and output data within a Calliope model updated to remove concatenated `loc::tech::carrier` sets and technology subsets (e.g. `techs_supply`) in favour of sparse arrays indexed over core dimensions only (`nodes`, `techs`, `carriers`, `timesteps`).

|changed| |backwards-incompatible| `coordinates.lat`/`lon` node parameter → `latitude`/`longitude`.

|changed| |backwards-incompatible| Distance units default to kilometres and can be reverted to metres with `config.init.distance_unit`.

|changed| |backwards-incompatible| `operate` mode input parameters now expected to match `plan` mode decision variable names (e.g., `flow_cap`).

|changed| |backwards-incompatible| Cyclic storage is defined per-technology, not as a top-level configuration option.

|changed| Documentation has been overhauled.

|removed| `_equals` constraints.
Use both `_min` and `_max` constraints to the same value.

|removed| `x`/`y` coordinates.
Use geographic lat/lon coordinates (in `EPSG:4326` projection) instead.

|removed| Comma-separated node definitions.
Inheriting duplicate definitions from `node_groups` instead.

|removed| `supply_plus` and `conversion_plus` technology base classes.
Use `supply` and `conversion` technology base classes instead.

|removed| `carrier` key.
Use `carrier_in` and `carrier_out` instead.

|removed| `carrier_tiers` and `carrier_ratios`.
Use indexed parameter definitions for `flow_out_eff` and your own math instead.

|removed| `calliope.Model.get_formatted_array`.
The Calliope internal representation of data now matches the format achieved by calling this method in v0.6.

|removed| Group constraints.
Use your own math instead.

|removed| Configuration options for features we no longer support.

|removed| Plotting.
See our documentation for example of how to visualise your data with Plotly.

|removed| Clustering.
Cluster your data before creating your Calliope model.
Mapping of timeseries dates to representative days is still possible.

Internal changes

|new| Automatic release uploads to PyPI and new accompanying pre-release pipeline.

|new| Choice of issue templates.

|new| YAML schema to catch the worst offences perpetrated in the model definition / configuration.
This schema is also rendered as a reference page in the documentation, replacing `defaults`/`config` tables.

|new| The model mathematical formulation (constraints, decision variables, objectives) is stored in a YAML configuration file: `math/base.yaml`.
Equation expressions and the logic to decide on when to apply a constraint/create a variable etc. are given in string format.
These strings are parsed according to a set of documented rules.

|changed| Development environment installation instructions (they're now simpler!).

|changed| Documentation has been ported to Markdown pages and is built using MKDocs and the Material theme.

|changed| Pre-processed model data checks are conducted according to a YAML configuration, instead of a hard-coded set of python functions.
An API will be created in due course to allow the user to add their own checks to the configuration.

|changed| Costs are now Pyomo expressions rather than decision variables.

|changed| When a model is loaded into an active session, configuration dictionaries are stored as dictionaries instead of serialised YAML strings in the model data attributes dictionary.
Serialisation and de-serialisation only occur on saving and loading from NetCDF, respectively.

|changed| Backend interface has been abstracted to enable non-Pyomo solver interfaces to be implemented in due course.

|changed| Repository structure has been re-configured to use the `src` layout, to rely on the `pyproject.toml` configuration file for most config, and to use only `.txt` requirements files (for pip+conda cross-compatibility)

|changed| CI moved from Azure pipelines (back) to GitHub Actions.

|changed| Stronger reliance on `pre-commit`, including a CI check to run it in Pull Requests.

|changed| |backwards-incompatible| Updated to Numpy v1.23, Pandas v1.5, Pyomo v6.4, Ruamel.yaml v0.17, Scikit-learn v1.2, Xarray v2022.3, GLPK v5. This enables Calliope to be installed on Apple Silicon devices, but changes the result of algorithmic timeseries clustering. [In scikit-learn version 0.24.0, the method of random sampling for K-Means clustering was changed](https://scikit-learn.org/0.24/whats_new/v0.24.html#changed-models). This change will lead to different optimisation results if using [K-Means clustering](https://calliope.readthedocs.io/en/v0.6.10/user/advanced_features.html#time-resolution-adjustment) in your model.

|changed| |backwards-incompatible| Removed support for Python version 3.7 since some updated dependencies are not available in this version.

|changed| Installation instructions for developers have changed since we no longer duplicate pinned packages between the development/testing requirements file (`requirements.yml`) and the package requirements file (`requirements.txt`). See [the documentation](https://calliope.readthedocs.io/en/v0.6.10/user/installation.html) for updated instructions.

|fixed| Set ordering in the model dataset is consistent before and after optimising a model with clustered timeseries. Previously, the link between clusters and timesteps would become mixed following optimisation, so running `model.run(force_rerun=True)` would yield a different result.