Bionemo

What's Changed

Applies fixes to ESM2 metric logging that result in `NotImplementedError` while using Model Parallelism.

**Full Changelog**: https://github.com/NVIDIA/bionemo-framework/compare/v2.4...v2.4.1

New Features

* Draft implementation of Evo2 with support for Hyena operators
* bionemo-moco v0.0.1 released for building diffusion-like generative models.

Updates & Improvements

* ESM2 fine-tuning script with CLI (finetune_esm2) that supports sequence-level/token-level classification/regression using a CSV dataset.
* [Brev.dev launchable fine-tuning tutorial for ESM2](https://nvidia.github.io/bionemo-framework/user-guide/examples/bionemo-esm2/finetune/)

What's Changed
* bump nemo and remove manual tensorstore install by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/619
* remove the apex and TE build steps from our docker container by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/611
* Adds bionemno-esm2 section to CODEOWNERS by jomitchellnv in https://github.com/NVIDIA/bionemo-framework/pull/627
* LR multiplier for ESM2 finetuning layers by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/609
* fix perplexity logging by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/622
* Allow finetuning ESM2 with [un]frozen encoder by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/620
* ESM-2 to NeMo checkpoint conversion by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/537
* remove PerplexityCallback in pydantic api by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/636
* catch ngc api key validation errors and default to not using an api key by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/635
* New approvals workflow by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/639

New Features

* Distributed Inference Support for ESM2 and Geneformer
* Enables linear inference throughput as GPU number is increased
* [See ESM2 inference notebook](https://github.com/NVIDIA/bionemo-framework/blob/release-v2.3/docs/docs/user-guide/examples/bionemo-esm2/inference.ipynb) and use `--num-gpus` parameter.

Updates & Improvements

* Prior Geneformer inference on H100 accuracy regression fixed.
* Base image updated to `nvcr.io/nvidia/pytorch:24.12-py3`; python updated to 3.12 among other core dependency upgrades ([base container release notes here](https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-24-12.html#rel-24-12)).

<details>
<summary><h3>Changes</h3></summary>

* Distributed Inference Support for ESM2/Geneformer by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/482
* Flexible memory management to avoid fragmentation-related CUDA OOM by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/524
* Update nightly Docker image tag by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/539
* set UV_NO_CACHE by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/529
* RowFeatureIndex Optimization by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/531
* Updates to NvFaidx, Fasta Noodles, and Sequence Accessor by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/532
* Fix csv dataset by holgerroth in https://github.com/NVIDIA/bionemo-framework/pull/543
* Run all pytests even if submodules fail by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/545
* xFail known bad tests on H100 and fix CVEs by gagank1 in https://github.com/NVIDIA/bionemo-framework/pull/549
* Fully Integrate SCDL into Geneformer by savitha-eng in https://github.com/NVIDIA/bionemo-framework/pull/480
* Fix MLM loss ignore idx by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/552
* Attempts to bump the base image to pytorch:24.07 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/544
* Pstjohn/update base image 2410 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/551
* [BUFIX] fail when passed fastas with duplicate sequence ids by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/555
* Update ddp config to improve ESM-2 15B MFU by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/520
* add temporary mistune pin to fix docs build issue by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/559
* Bump 3rdparty/Megatron-LM from `99f23d2` to `2da43ef` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/558
* Bump 3rdparty/NeMo from `06e6703` to `06a1491` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/538
* update base image to 24.12 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/553
* Un-xfail geneformer on H100 test by trvachov in https://github.com/NVIDIA/bionemo-framework/pull/563
* update devcontainer for new ubuntu base image by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/566
* don't eagerly download esm2 checkpoints by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/567
* run pytest with or without docs and notebooks in run_pytest.sh by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/569
* Jwilber/bionemo example small updates by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/561
* remove unused file from repo by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/562
* add initial configs for perf testing on ESM2 in JET (bionemo2) by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/497
* Add pre-training page for ESM-2 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/578
* Edits to README and CONTRIBUTING.md, moving some text around by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/577
* Refactor dockerfile for better caching and avoid pbss download in notebook test by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/573
* Bump 3rdparty/NeMo from `06a1491` to `d44ed44` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/580
* Simplify ESM2 finetune test by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/576
* default to overlap_param_gather by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/582
* Bump 3rdparty/Megatron-LM from `2da43ef` to `65720c8` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/579
* Add self-hosted azure runner workflows by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/587
* ARM docker build with 24.12 pytorch fw image by trvachov in https://github.com/NVIDIA/bionemo-framework/pull/581
* Add gpu target identificator to JET configs by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/586
* add codecov badge by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/588
* Add support for marking and skipping slow tests, temporarily mark pydantic tests as slow by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/589
* pin cdifflib version by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/593
* Remove outdated note on very large datasets in MultiEpochDataset by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/521
* Bump 3rdparty/NeMo from `eb9848b` to `abd4bf7` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/597
* Revert "pin cdifflib version (593)" by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/599
* fix esm2_pretrain.yaml by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/600
* add myself to ci by nvdreidenbach in https://github.com/NVIDIA/bionemo-framework/pull/594
* Bump virtualenv from 20.26.3 to 20.26.6 by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/596
* Bump 3rdparty/Megatron-LM from `65720c8` to `c76410a` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/592
* move load calls, rename test for better readibility by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/601
* Only run cleanup if tests ran, adds pytest marker config for slow tests by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/595
* only run trufflehog on diff by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/604
* run trufflehog on entire main branch on push action by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/605
* add comments to the unit-test.yaml file by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/606
* Remove v2.0 from README title by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/602
* ESM2 Finetuning refactor by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/574
* fix image links in esm2 model card by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/584
* Release of v1.0 of BioNeMo Modular Co-Design (MoCo) by nvdreidenbach in https://github.com/NVIDIA/bionemo-framework/pull/575
* fix devcontainer paths in ubuntu 24 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/610
* Bump rsync and other dockerfile lints by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/603
* Jm/codeowners revamp by jomitchellnv in https://github.com/NVIDIA/bionemo-framework/pull/617
* Update MoCo notebooks by nvdreidenbach in https://github.com/NVIDIA/bionemo-framework/pull/614
* set min seq len by default by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/621
* hotfix for some failing python tests due to NGC files being moved around by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/626
* Bump 3rdparty/Megatron-LM from `c76410a` to `4fb4c3d` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/624
* revert ESM2 Finetuning refactor (574) by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/628

New Contributors
* holgerroth made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/543
* nvdreidenbach made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/594

**Full Changelog**: https://github.com/NVIDIA/bionemo-framework/compare/v2.2...v2.3

* Disable validation/test stages in ESM-2 and Geneformer by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/492
* CI HOTFIX: ignore inrun_pytest.sh a notebook by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/526
* added NeMoLogger unit tests by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/511
* Bump 3rdparty/Megatron-LM from `844119f` to `99f23d2` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/528
* [cye/wandb-fix] Fix WandB issue. by cspades in https://github.com/NVIDIA/bionemo-framework/pull/530
* xFail known bad tests on H100 and fix CVEs by gagank1 in https://github.com/NVIDIA/bionemo-framework/pull/547

New Contributors
* sveccham made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/432
* sirelkhatim made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/447

**Full Changelog**: https://github.com/NVIDIA/bionemo-framework/compare/v2.1...v2.2

New Features:
* ESM2 Implementation
* Updated the ESM-2 Model Card with detailed performance benchmarks comparing BioNeMo2 training against vanilla pytorch.
* Added ESM-2 inference endpoint for evaluating pre-trained models
* Size-Aware Batching
* Added SizeAwareBatchSampler, a pytorch data sampler that batches elements of varying sizes while ensuring that the total size of each batch does not exceed a specified maximum.
* Added BucketBatchSampler, another pytorch data sampler that groups elements of varying sizes based on predefined bucket ranges, and create batches with elements from each bucket to ensure that each batch has elements with homogeneous sizes.
* CLI Support
* Added pydantic interface for pretraining jobs via parsing JSON configuration files that enables passing customized Model and DataModules classes.
* Implemented pydantic configuration for Geneformer and ESM2 pretraining and finetuning.
* Added 'recipes' for generating validated JSON files to be used with pydantic interface.
* Added installable scripts for 2/3 respectively, bionemo-esm2-recipe, bionemo-esm2-train, bionemo-geneformer-recipe, bionemo-geneformer-train.
* Geneformer support in BioNeMo2:
* Tested pre-training scripts and fine-tuning example scripts that can be used as a starting point for users to create custom derivative models.
* Geneformer 10M and 106M checkpoints ported from BioNeMo v1 into BioNeMo v2 available and included in documentation.
* Added inference scripts
* Documentation
* Cell type classification example notebook which covers the process of converting anndata into our internal format, and running inference on that data with a geneformer checkpoint, as well as making use of the inference results.
* Updated Getting Started guide, ESM-2 tutorials
* Added Frequently Asked Questions (FAQ) page


<details>
<summary><h2>Changes</h2></summary>

* Final October docs edits by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/331
* Update container location and tag for 2.0 release by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/337
* Remove broken Release Notes links from v2.0 docs build by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/343
* Tell pytest to ignore 3rdparty/{NeMo,MegatronLM} by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/61
* Add back the removed bionemo-core sub-package by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/25
* Fix bionemo-size-aware-batching, standardize pyproject.toml's & dependencies by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/284
* Add check bug fix label workflow by yzhang123 in https://github.com/NVIDIA/bionemo-framework/pull/250
* Adds geneformer overview by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/279
* Add ESM2 Dataset and Datamodule by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/78
* Test checkpoint IO loss is close to expected. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/37
* fix post-create command by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/152
* Drop dependency to internal docs by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/303
* Add initial configuration for mike (version management for docs) by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/330
* Update ESM2 model card with benchmarks by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/341
* Geneformer PEFT by gwarmstrong in https://github.com/NVIDIA/bionemo-framework/pull/155
* Update initialization in response to VDR by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/334
* Add GitHub workflow by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/9
* Reorganize bionemo-contrib into namespace packages by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/51
* Improve ESM2 pretraining tutorial from VDR feedback by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/336
* install geometric dependencies before invalidating caches with source copy by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/224
* ESM2 LoRA by gwarmstrong in https://github.com/NVIDIA/bionemo-framework/pull/218
* chown /usr/local's dist-packages to allow editing them in the devcontainer by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/111
* add search highlight + code copy capabilities by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/102
* ESM2 implementation by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/28
* Fix broken docs links on mike build by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/344
* Updates to Getting Started docs by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/179
* fix post-create command by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/88
* refactor doc structure and look by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/143
* Make ruff check pre-commit hook follow what CI does by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/201
* Add bionemo-gemoetric: A component library for PyTorch Geometric Models & Data by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/110
* [FEA] size-aware batching: a package for creating mini-batch in a memory consumption-aware manner by DejunL in https://github.com/NVIDIA/bionemo-framework/pull/168
* ESM2 Finetune bug fix and update by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/197
* add dev tools to devcontainer build by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/210
* places the ptl artifacts ignore lines to the root directory only. by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/21
* Jared/v2 main/nvidia styles by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/101
* rename bionemo-fw-ea to bionemo-framework by yzhang123 in https://github.com/NVIDIA/bionemo-framework/pull/292
* Add BERT-style masking function by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/55
* Add perplexity logging by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/144
* support nsys profiling on ESM2, add downstream improvements to hit P0 perf by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/300
* trivial commit to bionemo2 by broland-hat in https://github.com/NVIDIA/bionemo-framework/pull/19
* Add geneformer bionemo1 disclaimer by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/278
* Split out the lightning example tutorial by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/67
* Move v2 commits over. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/8
* Add documentation covering megatron and code structure rationalle by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/153
* try out gh page url to resolve 404 error by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/233
* lowercase file name so mkdocs picks up correctly by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/173
* use importlib resources for files by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/178
* add nemo-run as a git submodule by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/186
* Add module for loading test data. by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/120
* LightningDataModule for webdataset by DejunL in https://github.com/NVIDIA/bionemo-framework/pull/100
* Update dependency tags to match PR 36, and try to fix test failure by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/39
* Change to gelu default from relu which is what we actually used before by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/20
* Jwilber/load nb from subpackages by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/128
* Use github runners to run pre-commit hooks by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/42
* Bump 3rdparty/NeMo from `ff7c614` to `8f0d0c7` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/145
* Add a tested function to see if model parallel is enabled by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/175
* Handle special tokens in the bert masking function by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/99
* Fix all license headers to Apache by trvachov in https://github.com/NVIDIA/bionemo-framework/pull/347
* add dependabot file by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/161
* Checkpointing example with Geneformer by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/24
* epoch-level shuffling in ESM2 dataset by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/150
* Bump 3rdparty/Megatron-LM from `0bda578` to `08e80b0` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/183
* move CI scripts to central location by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/131
* setuptools sub-package local vs. publish by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/133
* Nested weight munging fine-tuning/continue training example and test for example model and geneformer. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/97
* ESM2 Golden Value Testing by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/85
* Add pretraining documentation by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/283
* Wandb integration by olachinkei in https://github.com/NVIDIA/bionemo-framework/pull/205
* Fix address in docs by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/297
* update branch name bionemo2 by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/160
* Updated README documentaiton for bionemo-{fw,core} by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/285
* Bump NeMo/Mcore by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/127
* Fix variable length ESM2 pretraining by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/306
* Establish CODEOWNERS for bionemo2 by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/121
* Revert "Jwilber/load nb from subpackages" by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/140
* add ci/scripts from ci repo - bionemo2 by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/214
* adding merge_group to existing actions by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/71
* Add license check to pre-commit hook by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/22
* ESM2 Fine-tune datamodule - epoch sampler by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/202
* Updates README by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/282
* Add some additional submodule commands to README by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/147
* Add documentation build system for BioNeMo v2 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/40
* Add future TE support and mixed precision support to biobert test by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/43
* add internal link for devcontainer cache by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/105
* Bump 3rdparty/NeMo from `0f8a531` to `a7d1896` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/172
* remove extra row in datasets by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/295
* Add a CLI option to restore training from a nemo1 checkpoint by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/54
* Fixing secuirty scan vulnerabilities by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/104
* Update devcontainer for new package structure by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/62
* add megatron datasets background, restructure background folder by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/237
* Pbinder/readme modify by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/115
* Adding Dataloading Test cases and documentation by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/107
* Bump 3rdparty/Megatron-LM from `cf0f9b2` to `ef85bc9` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/124
* Replace Launcher Script with Justfile & Standalone Scripts + Instructions for External by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/239
* Add some additional ruff checks, ignoring existing violations by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/56
* ESM2 Model Card by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/234
* FineTuning tutorial update [VDR] by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/342
* remove confest and make glob more specific by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/256
* Pbinder/move scdl by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/76
* fix infer_global_batch_size by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/261
* Adding license, and contributing guidelines from 72 and 65 by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/74
* Nemo 2 Model Checkpoint Load Test by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/270
* Reconfigure the pre-commit workflow by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/63
* ESM2 Finetuning README by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/240
* add tokenization test by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/169
* Add option to restore HF masking strategy by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/177
* Update README.md with marketing by ktretina in https://github.com/NVIDIA/bionemo-framework/pull/289
* Make artifact downloads more robust by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/41
* Updates to devcontainer by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/77
* Jwilber/fix docs dir creation by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/227
* refactor lightning module by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/123
* add jupyter notebook support in documentation by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/109
* Remove links in Overview Docs by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/338
* edits to make tests more amenable to being run against an installed package by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/154
* adding github action for docs deployment by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/98
* Bump 3rdparty/NeMo from `18d81b1` to `0f8a531` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/156
* Add mamba as a dependency in the dockerfile by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/44
* Fix DATA_DIR in esm2 pretraining by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/298
* Add trufflehog as a github action check by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/45
* package resource files with installed package by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/137
* Promote nltk version to address GHSA-cgvx-9447 by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/114
* bugfix: mkdocs-gen-files expects paths relative to the script's location by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/141
* multi-epoch dataset resamplers by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/174
* add docs test and remove duplicate pytest call by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/231
* moving test data around by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/118
* [cye/esm2-peft-tutorial] Add tutorial for ESM2 fine-tuning (training and inference), and PEFT training (but not inference). by cspades in https://github.com/NVIDIA/bionemo-framework/pull/263
* Fix Geneformer huggingface links by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/106
* adding some additional docstrings by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/81
* Bump 3rdparty/NeMo from `e6c0e72` to `ff7c614` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/103
* updating uniprot dataset card by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/200
* Complete ESM2 pretraining by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/112
* Migrate ESM2 to transformer engine by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/199
* Add NeMo and Megatron-LM as git submodules by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/52
* fixing devcontainer target by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/64
* Add check bug fix label workflow by yzhang123 in https://github.com/NVIDIA/bionemo-framework/pull/243
* move ESM2 dataset's odd rng call to use random_utils by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/280
* convert root_directory to a field with default_factory by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/58
* Add devcontainer config for bionemo2 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/5
* Adds CONTRIBUTING, CODE-REVIEW guides and pull request template by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/10
* Add-back `BioBertLightningModule` to fix model load bug by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/268
* Update Getting Started documentation to reflect BioNeMo2 workflow by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/208
* Update VERSION - release branch v2.0 by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/354
* Allow different model parallelism in pretrain/fine-tune or pretrain1/pretrain2 checkpoints. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/276
* Fine-tuning CLI example for geneformer by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/139
* Fix esm2 pp/tp by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/189
* Add Dorota + Peter as owners for ci scripts by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/166
* Jwilber/dark mode code color by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/252
* Add cc-by-4 attribution to cellxgene datacard by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/255
* adding better descriptions to bionemo-llm and bionemo-testing by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/222
* Pin ptl to <2.4.0 to fix nemo bug by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/86
* Add new paths for nemo2 checkpoints and update docs to use them by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/241
* Bump 3rdparty/Megatron-LM from `104d864` to `cf0f9b2` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/96
* ESM2 stop and go test by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/198
* Provide single team email address for authors in Python package metadata by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/167
* Bump 3rdparty/NeMo from `a7d1896` to `9ed0d6c` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/184
* Install test deps in release image, fix scdl example_notebook by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/221
* Refactor lightning module by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/235
* Clean up src vs test mirroring rule violations. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/66
* Fix tach modules & unpin dev version by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/299
* Upgrade nemo and megatron, and fix configs to reflect the change by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/92
* Fix ESM2 doc by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/291
* Make example notebook runnable in bionemo-scdl by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/130
* Use precision lowest value instead of -torch.inf by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/35
* Add back contributing & code review guidelines by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/142
* Jstjohn/fix geneformer multinode by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/17
* Resolve NaNs in ESM2 token-level fine-tuning loss by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/236
* Bump 3rdparty/Megatron-LM from `b6887d3` to `0bda578` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/171
* ESM2 finetuning by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/136
* streamline python packaging with uv by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/135
* Stop and Go harness and tests for geneformer and GPT. by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/116
* Revert "refactor lightning module" by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/217
* Megatron dataset compatibility checks by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/230
* add ngc url artefacts bionemo2 by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/254
* Make multi-line RUN statements fail fast by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/225
* Fixing the CLI for NGC paths that dump to stdout by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/271
* + Nvidia security policy document by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/163
* bump Megatron by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/148
* Pstjohn/release v2.0/releasenotes memleak by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/329
* some fixes to test builds by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/246
* fixing version issue by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/90

New Contributors
* skothenhill-nv made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/21
* farhadrgh made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/28
* polinabinder1 made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/76
* dependabot made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/96
* jwilber made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/101
* DejunL made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/100
* gwarmstrong made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/155
* olachinkei made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/205
* cspades made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/263
* ktretina made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/289
* guoqing-zhou made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/220
* savitha-eng made their first contribution in https://github.com/NVIDIA/bionemo-framework/pull/339

**Full Changelog**: https://github.com/NVIDIA/bionemo-framework/commits/v2.1

New Features:
* ESM2 implementation
* State of the art training performance and equivalent accuracy to the reference implementation
* 650M, and 3B scale checkpoints available which mirror the reference model
* Flexible fine-tuning examples that can be copied and modified to accomplish a wide variety of downstream tasks
* First version of our NeMo v2 based reference implementation which re-imagines bionemo as a repository of megatron models, dataloaders, and training recipes which make use of NeMo v2 for training loops.
* Modular design and permissible Apache 2 OSS licenses enables the import and use of our framework in proprietary applications.
* NeMo2 training abstractions allows the user to focus on the model implementation while the training strategy handles distribution and model parallelism.
* Documentation and documentation build system for BioNeMo 2.

Known Issues:
* PEFT support is not yet fully functional.
* Partial implementation of Geneformer is present, use at your own risk. It will be optimized and officially released in the future.
* Command line interface is currently based on one-off training recipes and scripts. We are working on a configuration based approach that will be released in the future.
* Fine-tuning workflow is implemented for BERT based architectures and could be adapted for others, but it requires you to inherit from the biobert base model config. You can follow similar patterns in the short term to load weights from an old checkpoint partially into a new model, however in the future we will have a more direct API which is easier to follow.
* Slow memory leak occurs during ESM-2 pretraining, which can cause OOM during long pretraining runs. Training with a
microbatch size of 48 on 40 A100s raised an out-of-memory error after 5,800 training steps.
* Possible workarounds include calling `gc.collect(); torch.cuda.empty_cache()` at every ~1,000 steps, which appears
to reclaim the consumed memory; or training with a lower microbatch size and re-starting training from a saved
checkpoint periodically.

External Partner Contributions

We would like to thank the following organizations for their insightful discussions guiding the development of the BioNeMo Framework and their valuable contributions to the codebase. We are grateful for your collaboration.

- [A-Alpha Bio](https://www.aalphabio.com/)
- [Dyno Therapeutics](https://www.dynotx.com/)
- [Relation Therapeutics](https://www.relationrx.com/)
- [Weights & Biases](https://wandb.ai/)

<details>
<summary><h2>Changes</h2></summary>

* Add GitHub workflow by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/9
* Move v2 commits over. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/8
* Jstjohn/fix geneformer multinode by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/17
* places the ptl artifacts ignore lines to the root directory only. by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/21
* ESM2 implementation by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/28
* Update dependency tags to match PR 36, and try to fix test failure by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/39
* Test checkpoint IO loss is close to expected. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/37
* Change to gelu default from relu which is what we actually used before by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/20
* Make artifact downloads more robust by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/41
* Add devcontainer config for bionemo2 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/5
* Add license check to pre-commit hook by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/22
* Use github runners to run pre-commit hooks by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/42
* Add back the removed bionemo-core sub-package by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/25
* trivial commit to bionemo2 by broland-hat in https://github.com/NVIDIA/bionemo-framework/pull/19
* Add mamba as a dependency in the dockerfile by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/44
* Add future TE support and mixed precision support to biobert test by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/43
* Add trufflehog as a github action check by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/45
* Adds CONTRIBUTING, CODE-REVIEW guides and pull request template by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/10
* Use precision lowest value instead of -torch.inf by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/35
* Add NeMo and Megatron-LM as git submodules by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/52
* Add a CLI option to restore training from a nemo1 checkpoint by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/54
* Add some additional ruff checks, ignoring existing violations by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/56
* Reorganize bionemo-contrib into namespace packages by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/51
* Update devcontainer for new package structure by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/62
* Tell pytest to ignore 3rdparty/{NeMo,MegatronLM} by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/61
* Clean up src vs test mirroring rule violations. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/66
* fixing devcontainer target by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/64
* adding merge_group to existing actions by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/71
* Split out the lightning example tutorial by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/67
* Reconfigure the pre-commit workflow by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/63
* convert root_directory to a field with default_factory by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/58
* Checkpointing example with Geneformer by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/24
* Updates to devcontainer by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/77
* Adding license, and contributing guidelines from 72 and 65 by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/74
* adding some additional docstrings by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/81
* Pin ptl to <2.4.0 to fix nemo bug by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/86
* Add documentation build system for BioNeMo v2 by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/40
* Add BERT-style masking function by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/55
* fix post-create command by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/88
* Pbinder/move scdl by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/76
* Add ESM2 Dataset and Datamodule by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/78
* Upgrade nemo and megatron, and fix configs to reflect the change by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/92
* Bump 3rdparty/Megatron-LM from `104d864` to `cf0f9b2` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/96
* ESM2 Golden Value Testing by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/85
* fixing version issue by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/90
* adding github action for docs deployment by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/98
* Jared/v2 main/nvidia styles by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/101
* Handle special tokens in the bert masking function by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/99
* add search highlight + code copy capabilities by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/102
* add internal link for devcontainer cache by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/105
* Fix Geneformer huggingface links by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/106
* Fixing secuirty scan vulnerabilities by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/104
* add jupyter notebook support in documentation by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/109
* Adding Dataloading Test cases and documentation by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/107
* Bump 3rdparty/NeMo from `e6c0e72` to `ff7c614` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/103
* Pbinder/readme modify by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/115
* Promote nltk version to address GHSA-cgvx-9447 by ohadmo in https://github.com/NVIDIA/bionemo-framework/pull/114
* moving test data around by polinabinder1 in https://github.com/NVIDIA/bionemo-framework/pull/118
* Bump 3rdparty/Megatron-LM from `cf0f9b2` to `ef85bc9` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/124
* Establish CODEOWNERS for bionemo2 by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/121
* chown /usr/local's dist-packages to allow editing them in the devcontainer by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/111
* Stop and Go harness and tests for geneformer and GPT. by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/116
* Bump NeMo/Mcore by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/127
* Complete ESM2 pretraining by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/112
* LightningDataModule for webdataset by DejunL in https://github.com/NVIDIA/bionemo-framework/pull/100
* Add module for loading test data. by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/120
* Jwilber/load nb from subpackages by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/128
* Nested weight munging fine-tuning/continue training example and test for example model and geneformer. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/97
* Make example notebook runnable in bionemo-scdl by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/130
* Revert "Jwilber/load nb from subpackages" by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/140
* package resource files with installed package by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/137
* move CI scripts to central location by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/131
* Fine-tuning CLI example for geneformer by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/139
* bugfix: mkdocs-gen-files expects paths relative to the script's location by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/141
* Add back contributing & code review guidelines by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/142
* setuptools sub-package local vs. publish by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/133
* Add some additional submodule commands to README by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/147
* bump Megatron by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/148
* fix post-create command by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/152
* epoch-level shuffling in ESM2 dataset by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/150
* Bump 3rdparty/NeMo from `ff7c614` to `8f0d0c7` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/145
* Bump 3rdparty/NeMo from `18d81b1` to `0f8a531` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/156
* edits to make tests more amenable to being run against an installed package by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/154
* update branch name bionemo2 by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/160
* + Nvidia security policy document by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/163
* Geneformer PEFT by gwarmstrong in https://github.com/NVIDIA/bionemo-framework/pull/155
* Provide single team email address for authors in Python package metadata by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/167
* Add bionemo-gemoetric: A component library for PyTorch Geometric Models & Data by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/110
* Add documentation covering megatron and code structure rationalle by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/153
* add dependabot file by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/161
* add tokenization test by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/169
* Bump 3rdparty/Megatron-LM from `b6887d3` to `0bda578` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/171
* refactor doc structure and look by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/143
* lowercase file name so mkdocs picks up correctly by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/173
* Bump 3rdparty/NeMo from `0f8a531` to `a7d1896` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/172
* Add a tested function to see if model parallel is enabled by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/175
* Add Dorota + Peter as owners for ci scripts by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/166
* use importlib resources for files by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/178
* Add option to restore HF masking strategy by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/177
* multi-epoch dataset resamplers by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/174
* Bump 3rdparty/NeMo from `a7d1896` to `9ed0d6c` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/184
* Bump 3rdparty/Megatron-LM from `0bda578` to `08e80b0` by dependabot in https://github.com/NVIDIA/bionemo-framework/pull/183
* ESM2 finetuning by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/136
* Fix esm2 pp/tp by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/189
* add nemo-run as a git submodule by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/186
* Updates to Getting Started docs by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/179
* ESM2 Finetune bug fix and update by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/197
* ESM2 stop and go test by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/198
* streamline python packaging with uv by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/135
* Add perplexity logging by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/144
* Make ruff check pre-commit hook follow what CI does by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/201
* refactor lightning module by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/123
* Revert "refactor lightning module" by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/217
* add dev tools to devcontainer build by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/210
* Migrate ESM2 to transformer engine by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/199
* ESM2 LoRA by gwarmstrong in https://github.com/NVIDIA/bionemo-framework/pull/218
* ESM2 Fine-tune datamodule - epoch sampler by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/202
* updating uniprot dataset card by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/200
* install geometric dependencies before invalidating caches with source copy by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/224
* adding better descriptions to bionemo-llm and bionemo-testing by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/222
* Install test deps in release image, fix scdl example_notebook by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/221
* add ci/scripts from ci repo - bionemo2 by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/214
* Make multi-line RUN statements fail fast by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/225
* [FEA] size-aware batching: a package for creating mini-batch in a memory consumption-aware manner by DejunL in https://github.com/NVIDIA/bionemo-framework/pull/168
* try out gh page url to resolve 404 error by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/233
* Wandb integration by olachinkei in https://github.com/NVIDIA/bionemo-framework/pull/205
* Jwilber/fix docs dir creation by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/227
* Resolve NaNs in ESM2 token-level fine-tuning loss by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/236
* add docs test and remove duplicate pytest call by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/231
* Add check bug fix label workflow by yzhang123 in https://github.com/NVIDIA/bionemo-framework/pull/243
* some fixes to test builds by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/246
* Jwilber/dark mode code color by jwilber in https://github.com/NVIDIA/bionemo-framework/pull/252
* Add new paths for nemo2 checkpoints and update docs to use them by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/241
* Megatron dataset compatibility checks by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/230
* add megatron datasets background, restructure background folder by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/237
* Add check bug fix label workflow by yzhang123 in https://github.com/NVIDIA/bionemo-framework/pull/250
* ESM2 Finetuning README by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/240
* remove confest and make glob more specific by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/256
* add ngc url artefacts bionemo2 by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/254
* Add cc-by-4 attribution to cellxgene datacard by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/255
* fix infer_global_batch_size by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/261
* Refactor lightning module by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/235
* [cye/esm2-peft-tutorial] Add tutorial for ESM2 fine-tuning (training and inference), and PEFT training (but not inference). by cspades in https://github.com/NVIDIA/bionemo-framework/pull/263
* Replace Launcher Script with Justfile & Standalone Scripts + Instructions for External by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/239
* ESM2 Model Card by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/234
* Add-back `BioBertLightningModule` to fix model load bug by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/268
* Nemo 2 Model Checkpoint Load Test by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/270
* Allow different model parallelism in pretrain/fine-tune or pretrain1/pretrain2 checkpoints. by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/276
* Fixing the CLI for NGC paths that dump to stdout by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/271
* Adds geneformer overview by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/279
* Updates README by skothenhill-nv in https://github.com/NVIDIA/bionemo-framework/pull/282
* Add pretraining documentation by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/283
* Update Getting Started documentation to reflect BioNeMo2 workflow by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/208
* Updated README documentaiton for bionemo-{fw,core} by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/285
* Fix bionemo-size-aware-batching, standardize pyproject.toml's & dependencies by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/284
* move ESM2 dataset's odd rng call to use random_utils by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/280
* rename bionemo-fw-ea to bionemo-framework by yzhang123 in https://github.com/NVIDIA/bionemo-framework/pull/292
* Fix ESM2 doc by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/291
* Fix address in docs by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/297
* remove extra row in datasets by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/295
* Update README.md with marketing by ktretina in https://github.com/NVIDIA/bionemo-framework/pull/289
* Fix DATA_DIR in esm2 pretraining by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/298
* Drop dependency to internal docs by farhadrgh in https://github.com/NVIDIA/bionemo-framework/pull/303
* support nsys profiling on ESM2, add downstream improvements to hit P0 perf by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/300
* Fix tach modules & unpin dev version by malcolmgreaves in https://github.com/NVIDIA/bionemo-framework/pull/299
* Add geneformer bionemo1 disclaimer by jstjohn in https://github.com/NVIDIA/bionemo-framework/pull/278
* Fix variable length ESM2 pretraining by sichu2023 in https://github.com/NVIDIA/bionemo-framework/pull/306
* Pstjohn/release v2.0/releasenotes memleak by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/329
* Add initial configuration for mike (version management for docs) by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/330
* Final October docs edits by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/331
* Update initialization in response to VDR by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/334
* Improve ESM2 pretraining tutorial from VDR feedback by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/336
* Update container location and tag for 2.0 release by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/337
* Remove links in Overview Docs by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/338
* FineTuning tutorial update [VDR] by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/342
* Update ESM2 model card with benchmarks by pstjohn in https://github.com/NVIDIA/bionemo-framework/pull/341
* Remove broken Release Notes links from v2.0 docs build by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/343
* Fix broken docs links on mike build by tshimko-nv in https://github.com/NVIDIA/bionemo-framework/pull/344
* Fix all license headers to Apache by trvachov in https://github.com/NVIDIA/bionemo-framework/pull/347
* Update VERSION - release branch v2.0 by dorotat-nv in https://github.com/NVIDIA/bionemo-framework/pull/354

**Full Changelog**: https://github.com/NVIDIA/bionemo-framework/commits/v2.0

Documentation and Field Support

Additional support and significant documentation overhauls performed by:

* isabel-wilkinson
* kkersten
* kushshah1
* NAEV95

</details>