Data-safe-haven

:warning: Update Requires Manual Intervention :warning:

If you are using a `4.1.0` SHM and want to upgrade to `4.2.0`, please follow the steps below:

1. Run `Setup_SHM_Firewall.ps1 -shmId {shmid}`
1. Run `Setup_SHM_Networking.ps1 -shmId {shmid}`
1. Delete `LINUX-UPDATES-SHM-{shmid}` VM and associated resources from the `RG_SHM_{shmid}_MONITORING` resource group
1. Delete `RG_SHM_{shmid}_PACKAGE_REPOSITORIES` resource group and all resources
1. Run `Setup_SHM_Update_Servers.ps1 -shmId {shmid}` (Note that this needs to happen before any further resources are deployed, since any further Linux resources will need access to the Linux update proxy).
1. Run `Setup_SHM_Package_Repositories -shmId {shmid}`
1. Run `Setup_SHM_Monitoring.ps1 -shmId {shmid}`

Known issues
* Jupyter notebook launched from GUI menu could not launch Python kernel, so it has been removed from the menu 065764734952ea776f26d331867301a7ddda7444

New Features
* Remove Microsoft Remote Desktop support: https://github.com/alan-turing-institute/data-safe-haven/pull/1535
* Remove CoCalc: https://github.com/alan-turing-institute/data-safe-haven/pull/1554
* Install dev dependencies in container: https://github.com/alan-turing-institute/data-safe-haven/pull/1747
* Add script to renew NFS share Stored Access Policies: https://github.com/alan-turing-institute/data-safe-haven/pull/1739
* Add script to automate account deletion: https://github.com/alan-turing-institute/data-safe-haven/pull/1508
* Factored out storage creation from SHM scripts https://github.com/alan-turing-institute/data-safe-haven/pull/1673
* SRD image updated, with latest Python versions available f3e890a4bc1010de60447c2f80db858c1e1a6197

Bug Fixes
* Update DBeaver drivers using Github workflow: https://github.com/alan-turing-institute/data-safe-haven/pull/1696
* Fixing DBeaver driver issues on T2+ SREs: https://github.com/alan-turing-institute/data-safe-haven/pull/1704
* Improve handling of spaces in file paths: https://github.com/alan-turing-institute/data-safe-haven/pull/1705
* Correct file path for Clam OnAccess scanning service: https://github.com/alan-turing-institute/data-safe-haven/pull/1725
* Fix PostgreSQL permissions and data schema, and relevant docs: https://github.com/alan-turing-institute/data-safe-haven/pull/1708
* Update outdated parameters that cause breaking change warnings: https://github.com/alan-turing-institute/data-safe-haven/pull/1663
* Change default lun from lun1 to lun0: https://github.com/alan-turing-institute/data-safe-haven/pull/1667
* Increase apt proxy server disk to 64 Gb: https://github.com/alan-turing-institute/data-safe-haven/pull/1726
* Remove `omsagent` from VM build image: https://github.com/alan-turing-institute/data-safe-haven/pull/1732
* Remove hyphens from SHM and SRE names in https://github.com/alan-turing-institute/data-safe-haven/pull/1650
* Update devcontainer configuration in https://github.com/alan-turing-institute/data-safe-haven/pull/1662
* Use memory for the /tmp directory in https://github.com/alan-turing-institute/data-safe-haven/pull/1672
* Remove unneeded opening bracket in SRE network configuration script https://github.com/alan-turing-institute/data-safe-haven/pull/1670
* Add missing import for logging module https://github.com/alan-turing-institute/data-safe-haven/pull/1681
* Fix `cloud-init` log parser using old name for event 58a85bc18368238cb2366fc5f77bb39944d5c1c8
* Detect and remove `omsagent` installed on SRD image before generalization e168b05b796e4123b9d7a8e98b0063c7abca7065

Security Fixes
* Update software on Guacamole and Nginx to latest versions: https://github.com/alan-turing-institute/data-safe-haven/pull/1741
* Update Nexus proxy server for T2/T3 package access: in https://github.com/alan-turing-institute/data-safe-haven/pull/1744
* Update CodiMD server version: https://github.com/alan-turing-institute/data-safe-haven/pull/1743
* Improve hardcoded domains and IP addresses: https://github.com/alan-turing-institute/data-safe-haven/pull/1745
* Prevent Nginx version information from appearing in http headers

Documentation updates
* Add guidance on resizing NFS shares: https://github.com/alan-turing-institute/data-safe-haven/pull/1749
* Update documents to reflect change to Microsoft Entra ID: https://github.com/alan-turing-institute/data-safe-haven/pull/1665
* Update deprecation warning for MS RDS: https://github.com/alan-turing-institute/data-safe-haven/pull/1542
* Add explanation of how to change allowed inbound IP addresses: https://github.com/alan-turing-institute/data-safe-haven/pull/1484
* Add all contributors table and instructions for how to update: https://github.com/alan-turing-institute/data-safe-haven/pull/1649
* Update contributors: https://github.com/alan-turing-institute/data-safe-haven/pull/1684
* Document removal of persistent SRE storage accounts: https://github.com/alan-turing-institute/data-safe-haven/pull/1685
* docs: update contributors: https://github.com/alan-turing-institute/data-safe-haven/pull/1686
* Add additional multiple data provider guidance to docs: https://github.com/alan-turing-institute/data-safe-haven/pull/1707
* Add links to guides for terminal, Xfce, and Guacamole: https://github.com/alan-turing-institute/data-safe-haven/pull/1737
* Update help text for Powershell command `shmId` and`sreId` arguments https://github.com/alan-turing-institute/data-safe-haven/pull/1683

**Full Changelog**: https://github.com/alan-turing-institute/data-safe-haven/compare/v4.1.0...v4.2.0

:warning: Update Requires Manual Intervention :warning:

If you are using a `4.X.Y` SHM and want to upgrade to `4.1.0`, please follow the steps below:

- Run `./deployment/safe_haven_management/setup/Setup_SHM_Networking.ps1 -shmId <your SHM ID>`
- Restart the virtual machine at `RG_SHM_<SHM name>_MONITORING/LINUX-UPDATES-SHM-<SHM name>` in the Azure portal

Known Issues

Only phone call authentication works for MS RDS. This provides no on-screen MFA Prompt.

New Features

* Allow device authentication in SHM deployment https://github.com/alan-turing-institute/data-safe-haven/pull/1378
* Add `arrow` CRAN package to Tier 3 core list https://github.com/alan-turing-institute/data-safe-haven/pull/1391
* Update Python in SRD images https://github.com/alan-turing-institute/data-safe-haven/pull/1421

Bug Fixes

* Update Powershell module requirements: 1368
* Update supported Powershell version to `7.3.6`
* Prevent removal of backup data during dry run: 1383
* Better package name matching for Nexus: 1447
* Update SRD image: 1421
* Add new servicebus endpoints for self-service password reset: 1423 and 1466
* Modify location of requirements.txt in Dockerfile: 1469
* Fixes of the SRD build related to python packages: 1514 and 1537
* Fix allowlist generation: 1422
* Update badges: 1371
* Update caching in allowlists workflow: 1395
* Fix incorrect logic around automated PR creation: 1426
* Update Ubuntu apt server addresses 1548
* Add docker.io to allowed-FQDNs 1548
* Change cloud-init files to automatically select appropriate disk partition 1548
* Fix MS-SQL database deployment 1580
* Fix PyPi Tier 3 mirror failures 1581

Security Fixes

* Fix non-allowed CRAN packages beginning with allowed name being installable: 1447
* Update to firewall rules: 1519

Documentation Updates

* Add instructions for installing documentation build dependencies: 1370
* Add instructions to resize VMs: 1367
* Update user management guide to explain adding users to security group and changing a phone number: 1389
* Add instructions for GPU VM resizing: 1399
* Add note on NVIDIA GPU support: 1406
* Remove reference to unused System Administrators Security Group: 1407
* Remove egress steps not carried out by System Manager: 1434
* Update SRE user troubleshooting: 1435
* Move from GitHub pages to ReadTheDocs 1468
* Add Policy for software package requests: 1387
* Add deprecation warning for MSRDS 1542
* Add warning that MSRDS does not work with the Microsoft Authentication app. 1589
* Add step for adding SSL certificate in step-by-step instructions for Guacamole 1590

**Full Changelog**: https://github.com/alan-turing-institute/data-safe-haven/compare/v4.0.3...release-v4.1.0

Bug fixes

- Update maximum allowed Powershell version
- Fix disk mounting issue when upgrading SRDs

Documentation updates

- Minor fixes

Bug fixes

- Add missing Powershell module imports
- Fix `-Upgrade` option when adding new SRD
- Fix `tensorflow` installation in SRD base image
- Register `Microsoft.DataProtection` on subscriptions that an SRE will be deployed into
- Support cross-subscription role assignments for backup
- Switch to correct subscription before deploying update automation
- Update Powershell version requirements to avoid upstream bug
- Update SRD package versions
- Use process-scope when retrieving Graph authorization tokens with Connect-MgGraph

Security fixes

- Remove unnecessary information from deployment logging

Documentation updates

- Add link to teardown docs to deployment page
- Add a VSCode `.devcontainer` for use in deployment
- Clarify that IP addresses are required in SRE config file
- Consolidate MFA setup description
- Update documentation build triggers to also run on `latest`

Bug fixes

- Add additional modules to requirements checker
- Add check for non-existing AzureAD security group
- Switch CI tests from Travis to GitHub Actions

Documentation updates

- Updated issue templates
- Fix documentation building

New features

- Add apt update server
- Add backup for blob storage
- Add backup for VM disks
- Add DNS server capabilities to DC2
- Enable automated VM updates
- Relicence to BSD 3-Clause
- Simplify deployment configuration
- Simplify NPS setup
- Simplify Powershell modules
- Switch to using DSC when configuring domain controllers
- Unify deployment of repository mirrors/proxies

Bug fixes

- Fix AAD domain verification
- Fix database logic so that either 0,1 or 2 databases can be deployed in an SRE
- Fix DNS recursion on domain controllers
- Fix htmlproofer issues by version pinning
- Fix network/firewall rules that were stopping the installation of gitlab-ce
- Fix NSG rules that were blocking LDAP connections from webapps
- Fix SHM teardown failure
- Fix Tier-3 allowlist scripts
- Fix updating of Guacamole dashboard when reading users from LDAP
- Improve tear down scripts
- Make RDS cipher suite setting more robust
- Make template deployments more robust
- Modify SHM requirements script to optionally install missing modules
- Restrict repository updates to this SRE
- Set Az.Storage minimum version
- Update NVIDIA repository key
- Update QGIS repository key
- Update SRD package versions
- Update to SSIS 16.0 in lockdown script

Security fixes

- Add ClamAV to all Linux VMs
- Drop support for Atom text editor
- Drop support for sbt
- Switch storage to GRS

Documentation updates

- Add administrator documentation for backups
- Add backup test to security checklist
- Add citation file
- Add disclaimer text to main repository README
- Add instructions to remove Conditional Access policies when reusing an AzureAD
- Add user backup instructions
- Fix various typographical errors in the documentation
- Make deployment instructions more visible
- Make documentation less prescriptive
- Update GitHub issue templates
- Update password writeback instructions
- Update SHM deployment instructions
- Update user guide