Data-safe-haven

Upgrading

This is a major release and it not compatible with any previous versions.
To use this version you must start a new TRE deployment.

Changes

- Complete rewrite of code in Python using IAC and configuration management tools Pulumi and Ansible

What's Changed

Known Issues

⚠️ This release is **not** ready for production usage. ⚠️

- ClamAV not configured
- Unstable container service IP addresses
- Lacking Nvidia utils

What's Changed
* Use pip-compile for package resolution by jemrobinson in https://github.com/alan-turing-institute/data-safe-haven/pull/1514
* Add pip-tools to NON_IMPORTABLE_PACKAGES by edwardchalstrey1 in https://github.com/alan-turing-institute/data-safe-haven/pull/1537
* Add May 2023 DSG to versioning by jemrobinson in https://github.com/alan-turing-institute/data-safe-haven/pull/1545

First version of migration to Python using Pulumi. Penetration tested in September 2023.

Known Issues

⚠️ This release is **not** ready for production usage. ⚠️

:warning: Update Requires Manual Intervention :warning:

If you are using a `4.2.x` SHM and want to upgrade to `4.2.2`, please follow the steps below:

For the SHM:
1. Add a `docker` section to your SHM config with a username and personal access token (following the SHM deployment instructions)
1. Re-run `Setup_SHM_Networking.ps1 -shmId {shm}` from `deployment/safe_haven_management/setup`

For any SRE that you deployed using an earlier `4.2.x` version:
1. Delete the `GUACAMOLE-SRE-{sreId}` VM and associated resources from the
`RG_SHM_{shmId}_SRE_{sreId}_REMOTE_DESKTOP` resource group
1. Re-run the deployment script `Deploy_SRE.ps1 -shmId {shm} -sreId {sre} -VmSizes {as before}` from `deployment/secure_research_environment/setup`

Known issues
- As for 4.2.0, 4.2.1

Bug Fixes

- Workaround for an issue where Let's Encrypt refused to provide certificates for uppercase FQDNs 1938
- Fix for change in Azure supported public IP address SKU for VPNs, which prevented deployment of the virtual network gateway for accessing domain controllers 1947
- Require supply of Docker Hub credentials to work round change in Docker download rate limits 1994
- Update approved IP address list for Ubuntu apt repositories
- Update to backup policy rules for Blob storage 1988

**Full Changelog**: https://github.com/alan-turing-institute/data-safe-haven/compare/v4.2.1...v4.2.2

:warning: Update Requires Manual Intervention :warning:

If you are using a `4.2.0` SHM and want to upgrade to `4.2.1`, please follow the steps below:

1. Delete the `GUACAMOLE-SRE-{sreId}` VM and associated resources from the `RG_SHM_{shmId}_SRE_{sreId}_REMOTE_DESKTOP` resource group
1. Re-run the deployment script `Deploy_SRE.ps1 -shmId {shm} -sreId {sre} -VmSizes {as before}` from `deployment/secure_research_environment/setup`

Known issues
- As for 4.2.0

Bug Fixes
- Update Guacamole to 1.5.5 to avoid [this known bug](https://lists.apache.org/thread/0sok6jgddhoxl01yvvlptqf1ptqnp5lc)

**Full Changelog**: https://github.com/alan-turing-institute/data-safe-haven/compare/v4.2.0...v4.2.1

:warning: Update Requires Manual Intervention :warning:

If you are using a `4.1.0` SHM and want to upgrade to `4.2.0`, please follow the steps below:

1. Run `Setup_SHM_Firewall.ps1 -shmId {shmid}`
1. Run `Setup_SHM_Networking.ps1 -shmId {shmid}`
1. Delete `LINUX-UPDATES-SHM-{shmid}` VM and associated resources from the `RG_SHM_{shmid}_MONITORING` resource group
1. Delete `RG_SHM_{shmid}_PACKAGE_REPOSITORIES` resource group and all resources
1. Run `Setup_SHM_Update_Servers.ps1 -shmId {shmid}` (Note that this needs to happen before any further resources are deployed, since any further Linux resources will need access to the Linux update proxy).
1. Run `Setup_SHM_Package_Repositories -shmId {shmid}`
1. Run `Setup_SHM_Monitoring.ps1 -shmId {shmid}`

Known issues
* Jupyter notebook launched from GUI menu could not launch Python kernel, so it has been removed from the menu 065764734952ea776f26d331867301a7ddda7444

New Features
* Remove Microsoft Remote Desktop support: https://github.com/alan-turing-institute/data-safe-haven/pull/1535
* Remove CoCalc: https://github.com/alan-turing-institute/data-safe-haven/pull/1554
* Install dev dependencies in container: https://github.com/alan-turing-institute/data-safe-haven/pull/1747
* Add script to renew NFS share Stored Access Policies: https://github.com/alan-turing-institute/data-safe-haven/pull/1739
* Add script to automate account deletion: https://github.com/alan-turing-institute/data-safe-haven/pull/1508
* Factored out storage creation from SHM scripts https://github.com/alan-turing-institute/data-safe-haven/pull/1673
* SRD image updated, with latest Python versions available f3e890a4bc1010de60447c2f80db858c1e1a6197

Bug Fixes
* Update DBeaver drivers using Github workflow: https://github.com/alan-turing-institute/data-safe-haven/pull/1696
* Fixing DBeaver driver issues on T2+ SREs: https://github.com/alan-turing-institute/data-safe-haven/pull/1704
* Improve handling of spaces in file paths: https://github.com/alan-turing-institute/data-safe-haven/pull/1705
* Correct file path for Clam OnAccess scanning service: https://github.com/alan-turing-institute/data-safe-haven/pull/1725
* Fix PostgreSQL permissions and data schema, and relevant docs: https://github.com/alan-turing-institute/data-safe-haven/pull/1708
* Update outdated parameters that cause breaking change warnings: https://github.com/alan-turing-institute/data-safe-haven/pull/1663
* Change default lun from lun1 to lun0: https://github.com/alan-turing-institute/data-safe-haven/pull/1667
* Increase apt proxy server disk to 64 Gb: https://github.com/alan-turing-institute/data-safe-haven/pull/1726
* Remove `omsagent` from VM build image: https://github.com/alan-turing-institute/data-safe-haven/pull/1732
* Remove hyphens from SHM and SRE names in https://github.com/alan-turing-institute/data-safe-haven/pull/1650
* Update devcontainer configuration in https://github.com/alan-turing-institute/data-safe-haven/pull/1662
* Use memory for the /tmp directory in https://github.com/alan-turing-institute/data-safe-haven/pull/1672
* Remove unneeded opening bracket in SRE network configuration script https://github.com/alan-turing-institute/data-safe-haven/pull/1670
* Add missing import for logging module https://github.com/alan-turing-institute/data-safe-haven/pull/1681
* Fix `cloud-init` log parser using old name for event 58a85bc18368238cb2366fc5f77bb39944d5c1c8
* Detect and remove `omsagent` installed on SRD image before generalization e168b05b796e4123b9d7a8e98b0063c7abca7065

Security Fixes
* Update software on Guacamole and Nginx to latest versions: https://github.com/alan-turing-institute/data-safe-haven/pull/1741
* Update Nexus proxy server for T2/T3 package access: in https://github.com/alan-turing-institute/data-safe-haven/pull/1744
* Update CodiMD server version: https://github.com/alan-turing-institute/data-safe-haven/pull/1743
* Improve hardcoded domains and IP addresses: https://github.com/alan-turing-institute/data-safe-haven/pull/1745
* Prevent Nginx version information from appearing in http headers

Documentation updates
* Add guidance on resizing NFS shares: https://github.com/alan-turing-institute/data-safe-haven/pull/1749
* Update documents to reflect change to Microsoft Entra ID: https://github.com/alan-turing-institute/data-safe-haven/pull/1665
* Update deprecation warning for MS RDS: https://github.com/alan-turing-institute/data-safe-haven/pull/1542
* Add explanation of how to change allowed inbound IP addresses: https://github.com/alan-turing-institute/data-safe-haven/pull/1484
* Add all contributors table and instructions for how to update: https://github.com/alan-turing-institute/data-safe-haven/pull/1649
* Update contributors: https://github.com/alan-turing-institute/data-safe-haven/pull/1684
* Document removal of persistent SRE storage accounts: https://github.com/alan-turing-institute/data-safe-haven/pull/1685
* docs: update contributors: https://github.com/alan-turing-institute/data-safe-haven/pull/1686
* Add additional multiple data provider guidance to docs: https://github.com/alan-turing-institute/data-safe-haven/pull/1707
* Add links to guides for terminal, Xfce, and Guacamole: https://github.com/alan-turing-institute/data-safe-haven/pull/1737
* Update help text for Powershell command `shmId` and`sreId` arguments https://github.com/alan-turing-institute/data-safe-haven/pull/1683

**Full Changelog**: https://github.com/alan-turing-institute/data-safe-haven/compare/v4.1.0...v4.2.0