Release notes
Alongside `yagna` 0.12, we’re releasing a new version - 0.10 - of `yapapi`, Golem’s Python API.
This new update comes with a set of both gradual improvements and important features that expand
the range of applications that can be developed on Golem and at the same time make the process
easier by adding to the toolset the developers have at their disposal.
New features
*Experimental* outbound network access
We’ve been aiming for some time to enable payloads to access external locations, knowing it would
open a vast world of new possibilities regarding the types of apps that can be built on Golem.
Therefore, one of the important features that `yagna` 0.12 and `yapapi` 0.10 introduce is the
**experimental** support for **outbound internet access** from Golem VM runtimes.
As we’re very consciously aware that giving access to all locations without any limits would make
Golem a dream-come-true platform for botnets of all kinds, the current release makes this feature
**experimentally available** in two flavours. Firstly, we’re adding a curated whitelist of locations
that the providers will have set up by default and which can be accessed by any VM payload.
Secondly, we’re enabling the VM images to be accompanied by application manifests that enable only
specific addresses to be accessed. Each of such manifests must be signed by the requestor's private
key and must carry a certificate that confirms the requestor has the privilege of running such apps on Golem.
Alongside the feature itself, comes a new example - `external-api-request` - that showcases the
latter type of usage.
This is the feature many of you have been asking about for a long time now and with it in place,
we cannot wait to see what kinds of new apps our community creators will be able to develop!
Generic TCP socket proxy
A few releases back, we have made the local http proxy available to enable HTTP-based services to be
accessed through a local port. In the meantime, it has become clear that other types of services
would also benefit from such access.
Consequently, in this release, we’re adding `SocketProxy` - a module that does exactly that, in
other words, it **exposes a local port** on a requestor’s machine and **forwards all the traffic**
to and from a remote port on a provider node through it.
In effect, **any type of service** - be it an SSH server, a remote SQL database or, say, redis
cache - as long as it’s a TCP service running inside a VM, can be very easily accessed **through
a port on the requestor’s machine** without any additional, intermediary tools.
We have also updated our SSH example to show the usage of this feature.
This does not yet enable direct access to those ports on the provider node’s IP addresses and
this is a feature we’ll make available in some future release.
Minor features
IP address removal and recycling
While our initial support for Golem VPN allowed a user to define the IP addresses of launched
services, it only allowed those addresses to be assigned once when a given service instance was
started.
Even though it works well in a happy-path scenario when each of the commissioned services launches
successfully the first time around, it becomes problematic when there is a need to restart an
instance, either because we want to modify it or just because it failed to initialise properly, and
we want to relaunch it on another provider node.
The new update enables the user to explicitly remove the IP address mapping for each of the nodes in
the network, but it also **automatically frees all the IP addresses** when instances are terminated.
Additionally, failing services with statically-assigned IPs are now able to be restarted correctly.
More convenient Service restart hooks
In the previous release, we have added the `Service.reset` hook that allowed the application authors
to define operations needed for `Service` to be brought back to an initial state when an instance
was about to be restarted.
That said, the engine didn't give the user a convenient way to define the exact scenarios
*when* instances should be restarted - the previous, `respawn_condition` hook was not easily
available outside the API itself.
Because of that, we have decided to define an explicit location to enable such customised behaviour,
namely the `Service.restart_condition` property. The default condition is consistent with the
previous, built-in behaviour of the engine which triggers a restart only if the instance it had not
been successfully started. Application authors can easily now override this default to provide
their own, **tailored criteria for restart**.
Customised debit note and invoice acceptance criteria
Up until now, the requestor agent always accepted the full amount specified in the invoices and
debit notes coming from the providers.
The new update adds the `debit_note_accepted_amount` and `invoice_accepted_amount` hooks to
`MarketStrategy`, enabling the requestor agent authors to define their own criteria and calculations
that determine whether incoming debit notes and invoices are accepted and whether they should be
accepted in full or partially.
Automatic service activity health-check
Previously, the requestor agent had no way of knowing that a given service stopped being available,
unless the launched Service instance actively interacted with the underlying activity running on the
provider node.
In this release, we have added a background task periodically querying the activity for its state.
Thanks to that, the requestor will be notified of a given service’s failure soon after it happens.
This allows the application authors to program recovery behaviour into their code more efficiently.
Full changelog
New features
* Add **experimental** support for manifest payloads (991 / 994 / 1008)
* Add a new `external-api-request` example that uses the manifests payload to enable outbound traffic
* Add `yapapi.contrib.SocketProxy` - a generic proxy to facilitate connections to any TCP-based services running on providers (1027)
* Remove dependency on `websocat` in the `ssh` example
* Add support for removal of IP addresses from a VPN (1054)
* Recycle IP addresses of failed Service instances so they can be restarted on a new node (1054)
* Enable `Service` to implement logic to decide whether instances should be restarted (1022)
* new `Service.restart_condition` hook
* Allow `MarketStrategy` classes to define their own criteria for debit note and invoice acceptance (954)
* new `MarketStrategy.invoice_accepted_amount` hook
* new `MarketStrategy.debit_note_accepted_amount` hook
* Add a health-check task to continually verify the state of running Service instances and detect defunct ones (1033)
* Add `Service.is_activity_responsive` method
Backward-incompatible changes
* Remove components deprecated in 0.6 and 0.7 (919)
* Remove `driver` and `network` arguments of the `Golem` class
* plus the respective `driver` and `network` properties
* remove `yapapi.package.vm` module (use `yapapi.payload.vm` instead)
* Remove exescript helpers from `yapapi.ctx` and make usage of the `Script` explicit:
* `yapapi.ctx.CommandContainer`
* `WorkContext.commit`
* command methods from `WorkContext`, e.g. `ctx.run()`
* client code should just yield `Script` objects from task / service handlers instead
* Remove `yapapi.executor.ctx`, `yapapi.executor.events`, `yapapi.executor.strategy`
* these had been moved to `yapapi.engine`
* Remove `yapapi.props.inf.RuntimeType` enum,
* Use `yapapi.props.inf.RUNTIME_*` constants directly
* Remove `yapapi.props.inf.InfVm`,
* those properties had been included into `yapapi.payload.vm`
* Replace `ServiceStarted` event with `ServiceStateChanged` (958)
Minor updates
* Change the default subnet to `public` (1048)
Fixes
* Capture and handle errors in `Strategy.score_offer` (908)
* Enable the `LocalHTTPProxy` to properly handle requests larger than 64kbytes (1013)
* Add a modified example web app example that uses file uploads to test larger requests
* Allow the engine to accept one additional `DebitNote` when activities are terminated (1030)
* Add handling for remote response timeouts in the local http proxy (1050)
* Fix handling and reporting of IP address collisions in the VPN (1053)
Internal changes
* Overhaul the invoice processing and introduce `InvoiceManager` to encapsulate all logic related to processing of invoices from the providers. 886
* Add concurrent `DebitNote` handling (902)
* Split proposal-receiving and proposal-scoring (968)
* Use updated `yagna` REST API bindings (944)
* Copy `distutils.util.strtobool` to `yapapi.utils` because of deprecation (1009)
* Remove pre-commit configuration and update the README (1018)
* Add `isort` to project dependencies and checks to enforce imports ordering (1034)
CI and Testing
* Add an integration test for the `webapp` example (974)
* Add an integration test for the `scan` example (977)
* Update `goth` to newest version
* Various minor fixes improving the reliability of the integration tests