Pg-spot-operator

-------------------
- Release: version 0.9.46 🚀 [Kaarel Moppel]
- Add a Lifecycle column to the --list-instances output. [Kaarel Moppel]

To show if it's a spot or on-demand instance. Also show the correct
on-demand price for the on-demand instances
- Allow VM provisioning with --ssh-private-key pointing to a PEM file.
[Kaarel Moppel]

Currently we checked for .pub existance
- Simplify aws ec2 authorize-security-group-ingress example. [Kaarel
Moppel]

To expose an SSH port for Ansible to work
- Male fail2ban less aggressive. [Kaarel Moppel]

5 > 100 errors per 1h
- Show help again if no CLI arguments set. [Kaarel Moppel]

Got messed up with default --vm-login-user set to pgspotops
- Fix persistent VM (--persistent-vms) price sorting. [Kaarel Moppel]

Was random essentially
- Add a special -1 value to --storage-min handling. [Kaarel Moppel]

Don't create / attach a data volume in that case and only run with root
volume. Default --os-disk-size is 20GB.

-------------------
- Release: version 0.9.45 🚀 [Kaarel Moppel]
- README minor correction on boolean flags. [Kaarel Moppel]
- Merge pull request 136 from pg-spot-ops/check-ssh-connectivity-after-
vm-provisioning. [Kaarel Moppel]

Check SSH connectivity not just wait 10s after VM provisioning and before Ansible setup, so that the first loop would always succeed
- A time based SSH connectivity check loop instead count based. [Kaarel
Moppel]

For better predictibility
- Check SSH connectivity after VM provisioning and before Ansible setup.
[Kaarel Moppel]

To avoid the occasional connectivity errors on the first loop, as
hard-coded 10s or 15s doesn't always seem to work.

As per https://github.com/pg-spot-ops/pg-spot-operator/issues/124
- Security README - mention current secrets handling posture /
assumptions. [Kaarel Moppel]

Close for now https://github.com/pg-spot-ops/pg-spot-operator/issues/62
- Merge pull request 135 from pg-spot-ops/record-app-user-password-in-
pgpass. [Kaarel Moppel]

Store app user password in postgres user .pgpass if set to be able to look it up later

As per https://github.com/pg-spot-ops/pg-spot-operator/issues/129
- Formatter bycatch. [Kaarel Moppel]
- Make .pgpass setting a bit nicer. [Kaarel Moppel]

Respect possible user modifications if any
- Store app user password in postgres user .pgpass if set. [Kaarel
Moppel]

To be able to look it up later. As per https://github.com/pg-spot-ops/pg-spot-operator/issues/129
- Merge pull request 134 from pg-spot-ops/enable-huge-pages-for-large-
ram-skus. [Kaarel Moppel]

Enable Linux huge pages for larger (>8GB) shared_buffers
- Stay on default 'huge_pages=try' for a test period still. [Kaarel
Moppel]

Instead of 'on' which fails if not enough huge pages available
- Enable huge pages for larger (>8GB) Shared Bufffers. [Kaarel Moppel]

As can bring visible performance improvements.

https://github.com/pg-spot-ops/pg-spot-operator/issues/132
- Merge pull request 133 from pg-spot-ops/disable-transparent-huge-
pages-on-boot. [Kaarel Moppel]

Disable Linux Transparent Huge Pages directly after boot
- Disable Linux Transparent Huge Pages directly after boot. [Kaarel
Moppel]

As generally negative for databases that manage their own memory
- Amend last. [Kaarel Moppel]

Fix circular import not catched by linter
- Keep trying other instance types when Spot vCPU count exceeded.
[Kaarel Moppel]

As limitations could apply only to a specific instance family
- Merge pull request 131 from pg-spot-ops/add-postgres-monitoring-via-
pgwatch. [Kaarel Moppel]

Add postgres monitoring via pgwatch
- Ensure Grafana service started on boot. [Kaarel Moppel]
- Add some pgwatch Prom dashboards. [Kaarel Moppel]
- Add Postgres monitoring via pgwatch. [Kaarel Moppel]

Using pgwatch 3.0 release from https://github.com/cybertec-postgresql/pgwatch/releases

Also adding local trust auth by default so that pgwatch can already
connect seamlessly during the setup
- Add Python versions badge. [Kaarel Moppel]

-------------------
- Release: version 0.9.41 🚀 [Kaarel Moppel]
- Add a CodeCov badge. [Kaarel Moppel]
- Set CodeCov to "informational" mode. [Kaarel Moppel]
- Merge pull request 127 from pg-spot-ops/add-codecov-config. [Kaarel
Moppel]

Add codecov.yml
- Add codecov.yml. [Kaarel Moppel]

Don't force quotas for now
- Run the setup finished callback also in --vm-only mode. [Kaarel
Moppel]

Feed VM IP addresses as input instead of PG connstr though
- Degrade instance destroy signal file path msg to debug. [Kaarel
Moppel]
- Warn if ansible-playbook executable not found on the PATH. [Kaarel
Moppel]

and if full Postgres setup mode
- Don't require setting of --storage-min for --list-instances to work.
[Kaarel Moppel]
- Connstr outputting improvements. [Kaarel Moppel]

Add a new "auto" --connstr-format and respect "postgres" format even when --vm-only set

-------------------
- Release: version 0.9.40 🚀 [Kaarel Moppel]
- Merge pull request 126 from pg-spot-ops/revert-boolean-cli-flag-
handling. [Kaarel Moppel]

Revert to original boolean CLI flag input handling + rename 2 flags to accomodate that

Don't require true/yes input for booleans
- Rename --ip-floating to --static-ip-addresses. [Kaarel Moppel]

For CLI convenience
- Rename --assign-public-ip to --private-ip-only. [Kaarel Moppel]

So that it works better via Py CLI
- Revert ba6518633: Change CLI input flags logic - all boolean flags
accepting values now. [Kaarel Moppel]

As quite inconvenient and non-pythonic in practice and given most people
will probably go for Docker usage anyways
- Some more README readability improvements. [Kaarel Moppel]
- Adjust network NVme random_page_cost to 1.25 from 1.1. [Kaarel Moppel]
- Set default_toast_compression to 'lz4' [Kaarel Moppel]

Replacing postgres default pglz as seems to be overall better:
https://www.timescale.com/blog/optimizing-postgresql-performance-compression-pglz-vs-lz4
- README - add a beta status label. [Kaarel Moppel]
- --list-regions - mention that not all regions might be actually
usable. [Kaarel Moppel]

And add an AWS CLI command to list enabled regions
- Show the monthly spot price + AZ when re-trying a launch. [Kaarel
Moppel]

As in case of not enough Spot capacity the the next tried price / zones can
already be different
- Set --ram-min default to 1GB. [Kaarel Moppel]

As 512MB instances occasionally throw OOM during Ansible setup

-------------------
- Release: version 0.9.36 🚀 [Kaarel Moppel]
- Increase MAX_SKUS_FOR_SPOT_PRICE_COMPARE from 15 to 25. [Kaarel
Moppel]

To cover case where cheaper burstable instances did not get selected as
they had more cores, when --allow-burstable set
- Expose the vm.allow_burstable attribute on the CLI. [Kaarel Moppel]

As --allow-burstable. Disabled by default
- Add cloud / SKU details to Ansible inventory output. [Kaarel Moppel]

Effective for --connstr-output-path + --connstr-format=ansible

Outputs now smth like: cat ans1.ini
51.92.207.184 ansible_user=pgspotops ansible_ssh_common_args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
; cloud=aws region=eu-south-2 provider_id=i-038f8779f966b8d73 sku=c6gd.medium
- Warn if no SSH access keys provided / found in real modes. [Kaarel
Moppel]

Proceed though as might be a test run still
- Docs - add an example date to --expiration-date. [Kaarel Moppel]

As ISO 8601 might require googling for most people
- Consider only enabled regions by default for price check. [Kaarel
Moppel]

And --list-instances, if --region not set explicitly. Fall back to all
known regions if describe_regions call fails
- README document that CLI assumes a local Ansible installation. [Kaarel
Moppel]

To be able to actually configure Postgres
- CLI - show help when no action params set. [Kaarel Moppel]

Also remove the --show-help param as not needed anymore

-------------------
- Release: version 0.9.35 🚀 [Kaarel Moppel]
- Merge pull request 122 from pg-spot-ops/readme-up. [Kaarel Moppel]

Readme update - Rework TOC, move to header.
- Rework TOC. [Kaarel Moppel]
- README - highlight the AI / GPU use case. [Kaarel Moppel]

Under Not only for Postgres
- README: mention the new --list-avg-spot-savings flag. [Kaarel Moppel]

And tidy the --list-avg-spot-savings output a bit
- Merge pull request 121 from pg-spot-ops/regional-spot-stats-summary.
[Kaarel Moppel]

New flag `--list-avg-spot-savings` to show regional spot savings and eviction rate summary
- Add a Mean Time to Eviction column to --list-avg-spot-savings output.
[Kaarel Moppel]
- Rename flag to --list-avg-spot-savings. [Kaarel Moppel]

And leave out the avg vCPU price as hard to calculate reliably
- Show correct avg. saving rate + ljustify. [Kaarel Moppel]
- Rename new flag to --regional-spot-stats. [Kaarel Moppel]

Plus table output
- New CLI flag to get avg Spot pricing / eviction rate stats. [Kaarel
Moppel]

Skel
- Random selection strategy needs an extra price sort. [Kaarel Moppel]

Even in single region search
- New flag: --ram-max / RAM_MAX. [Kaarel Moppel]

To limit cost for eviction-rate strategy, as lowest eviction rate
bracket machines could get huge and expensive
- Set Ansible default root to ~/.pg-spot-operator/ansible. [Kaarel
Moppel]

And make dev location secondary