Release notes
We're happy to announce a major release of yapapi - Golem's Python API 0.9.
Events
This update adds a completely overhauled [events subsystem](https://yapapi.readthedocs.io/en/latest/api.html#module-yapapi.events) that makes it much easier to implement more complicated functionality on top of `yapapi`. Additionally, it allows those components to be better decoupled and more independent from each other, increasing their composability and reusability.
Also, the new events now carry all the objects they pertain to instead of just the identifiers of those objects, as was the case previously. This saves the developers the hurdle of having to take care of storing, handling and retrieving those objects in their own code.
On top of that, the new events interface allows the developers to easily write event consumers that listen only to certain types of events and which can also handle custom events defined outside of yapapi.
Immediately available Service instance handlers
Another important addition to the API is the ability to get handles to Service instances immediately after those instances are requested through a `Golem.run_service` call.
Thanks to this change, the client code can start explicitly tracking what is happening with the commissioned Service instances without having to "guess" their status.
General motivation behind this change was better management of the services deployed through yapapi and simplification of code that wishes to extend this functionality further.
Yapapi Contrib
We have decided to introduce a new part of yapapi, aimed less at implementing the functionality of the library itself and more at adding additional components that are build on top of yapapi.
This part of yapapi repository serves to provide requestor agent application authors with reusable components that we or third-party developers found useful while working on examples and apps that use our Python API.
They’re not strictly part of the high level API library itself and while we do intend to keep them in sync with the subsequent releases, they should be treated as experimental and their interface may change without a proper deprecation process.
The first components that you can already use are the ProviderFilter market strategy helper and an http proxy component for network-enabled services.
ProviderFilter
ProviderFilter is a market strategy wrapper that enables easy exclusion of offers from certain providers using a simple boolean condition, while preserving correct scoring of the remaining offers by the base strategy.
It can be used to construct your own custom deny lists based on a simple registry or it can be extended through the new events interface to construct such a deny list dynamically based on some events (e.g. failure of an activity on a specific provider node).
LocalHTTPProxy
This component was previously part of the [HTTP proxy example](https://github.com/golemfactory/yapapi/tree/master/examples/http-proxy) which we made available in our last major release.
Now, we have extracted it and we're making it available to all yapapi developers as a stand-alone module that you can include in the apps that you build on top of yapapi.
The local HTTP proxy enables easy connections to any VPN-enabled, HTTP-based services launched on Golem providers using yapapi’s Services API.
Accompanying it is a new example - a [web application](https://github.com/golemfactory/yapapi/tree/master/examples/webapp) composed of a simple web server and a separate database node.
Mid-agreement payments
To allow longer agreement times, mainly in order to enable services running on Golem for durations of e.g. weeks and months as opposed to minutes or hours, we have added the mid-agreement payments support which works with the analogous update to newest version of yagna's provider agent.
This functionality relies on additional negotiation between the providers and requestors, through which the requestor agrees to pay the provider in set intervals within the duration of an agreement.
Before this change, the payments had only been made after an invoice was issued at the end, when the agreement has been terminated. Now, the agreement can persist indefinitely as long as the provider stays active and as long as the requestor continues to pay for that activity.
[More on that in our handbook.](https://handbook.golem.network/payments/payments-explained#mid-agreement-payments).
Early break from Task API's worker function
The last small but important feature we're making available in this release in yapapi is the ability to cleanly exit from Task API's worker function when new tasks are still available.
We've found this useful in various scenarios where, for any reason, we don't want to just spread our task over the Golem network to distribute the computational load but are rather focused on the executing nodes themselves.
This might include needs such as e.g. running a benchmark on as many nodes as possible to construct a list that we later use to run proper computations, or running a task that takes advantage of geographical distribution of Golem's nodes.
This feature is presented through our [new node scanner example](https://github.com/golemfactory/yapapi/blob/master/examples/scan).
New examples
As already mentioned above, we've prepared two new examples showcasing new yapapi features.
Node scanner
The [node scanner](https://github.com/golemfactory/yapapi/tree/master/examples/scan) example presents a clean and simple way of using yapapi's Task API to run a single task per a provider node to find what CPU brand is each provider using.
While not very useful on its own, it serves as a template for any scenario where a single (or a specific number of) task on a single node is needed - or - any scenario where we'd like to stop executing tasks on a specific node for whatever other reason.
Web application
The last but not the least, we're presenting you with [a web application](https://github.com/golemfactory/yapapi/tree/master/examples/webapp) example, which utilizes the Service API, yagna's VPN capability alongside the associated Network API in yapapi and finally the aforementioned Local HTTP Proxy module to run a proper web application on Golem.
The app is composed of an extremely simple HTTP server written in Python and using Flask and SQLAlchemy and of a database server running on rqlite (which is a replication-capable db back-end based on SQLite).
Both components are running on separate provider nodes and are connected by a virtual private network (VPN) both to themselves and to the requestor node.
Finally, thanks to the [local HTTP proxy](https://yapapi.readthedocs.io/en/latest/api.html#local-http-proxy), we're able to open a port on requestor's machine and route all traffic from that port to the HTTP server on a provider.
Miscellanea
Apart from that, we're including bunch of other, minor updates including fixes and small improvements here and there. Please have a look at the detailed changelog below.
Detailed changelog
Backward-incompatible changes
* Any functionality relying on the API's events will need to be updated to use the new events instead.
Major updates
* Overhauled events subsystem - [Events API Reference](https://yapapi.readthedocs.io/en/latest/api.html#module-yapapi.events)
* Add Golem.add_event_consumer method (755)
* ComputationStarted/Finished events in services API (761)
* Services now emit Service[Started|Finished] instead of Task[Started|Finished] (790)
* Replace PaymentAccepted with (Invoice|DebitNote)Accepted (792)
* Emit events using actual objects of interest instead of just their ids (783)
* New events, with full objects as attributes (e.g. agr_id -> Agreement) (801)
* Change the exception interface for events (798)
* Improve event-related type hints (802)
* Add custom repr and str methods to Event class (806)
* Event-based DecreaseScoreForUnconfirmedAgreement strategy (805)
* Add reprs to script event objects
* General cleanup of the Events-related functionality (840)
* Event-related cleanup of `yapapi.log.SummaryLogger` (847)
* `Golem.add_event_consumer` is no longer `async`. (866)
* Start consumers only when Golem is operative
* Event pre-filtering in `Golem.add_event_consumer`. (868)
* Separate ServiceRunner from the Cluster and expose immediately available handles to Service instances (750)
* `yapapi.services` split to separate files
* `yapapi.services.Cluster` split to `ServiceRunner` and a `Cluster`
* Add `Service.reset()` method, called when a service is restarted, since handlers of restarted instances can be called more than once now
* Introduce `yapapi.contrib` - a helper library of reusable components
* New `ProviderFilter` helper strategy (811) - see [ProviderFilter API reference](https://yapapi.readthedocs.io/en/latest/api.html#provider-filter)
* Make the local http proxy a reusable component (860) - see [LocalHttpProxy API reference](https://yapapi.readthedocs.io/en/latest/api.html#local-http-proxy)
* make the proxy more robust
* better error handling in HttpProxyService
* Implement the mid-agreement payments mechanism enabling indefinitely long agreements between the providers and requestors (831) and (864) - [ **TODO** add a link to the new Payments article ], [GAP-3 Mid-agreement payments](https://github.com/golemfactory/golem-architecture/blob/master/gaps/gap-3_mid_agreement_payments/gap-3_mid_agreement_payments.md)
* Add a general mechanism to handle negotiable properties
* add a new WrappingMarketStrategy interface and convert all current wrapping strategies to `WrappingMarketStrategy` (877) - [**TODO** add a link to the API reference]
* Provide a clean way of an early break from a worker function (836)
* plus, ensure events and logs are emitted correctly
Minor updates
* Add `Agreement` to `WorkContext` (781)
* Add type hints for service instances spawned within a cluster (793)
* Fix `Cluster`'s `__repr__`
* Add network info to Service instance's repr (860)
* rename Agreement `details` and `cached_details` to `get_details` and `details` respectively
* Default strategy moved from Engine to Golem
Documentation
* Add [Golem.start()](https://yapapi.readthedocs.io/en/latest/api.html#yapapi.Golem.start)/[Golem.stop()](https://yapapi.readthedocs.io/en/latest/api.html#yapapi.Golem.stop) to the API Reference (768)
* [Events API Reference](https://yapapi.readthedocs.io/en/latest/api.html#module-yapapi.events) (764)
* Custom events documentation (897) (898)
* Add an introduction to `yapapi.contrib` (899) - [yapapi.contrib API reference](https://yapapi.readthedocs.io/en/latest/api.html#module-yapapi.contrib)
* docs for yapapi.contrib.service.LocalHttpProxy/HttpProxyService (901) - [LocalHttpProxy/HttpProxyService API reference](https://yapapi.readthedocs.io/en/latest/api.html#local-http-proxy)
* remove redundant information from README.md, add more about goth testing (795)
Bugfixes
* Recycle offer from a rejected agreement (765)
* Don't accept payable debit notes at all if mid-agreement payments had not been negotiated
* Bump the `attrs` requirement to 0.19 to support the kwargs argument (914)
Deprecation, cleanups
* Remove _cli package and its dependencies (602)
* Deprecate `yapapi.strategy.DummyMS`
Examples
New examples
* New web application example - using two different service types to run a simple web application on Golem (860)
* New scanner example, presenting the proper way of running exactly one task per a provider node (836)
Fixes to examples
* Generator instead of a list in the hashcat example (804)
* Set min-cpu-threads to 1 in the blender example (757)
* Ensure `start_stop_blender.py` example pays invoices on ctrl+c (885)
Unit and integration tests
New tests
* Add an integration test for the SSH example (819)
* Mid-agreement payments integration test (883)
* Tests for `MarketStrategy` and `WrappingMarketStrategy` (887)
* Tests for the DecreaseScoreForUnconfirmedAgreement strategy
* Tests for the Mid-agreement payments functionality and associated changes in the market strategies (888)
* add Agreement factories
* add Debitnote factories
Fixes
* Increase PROPOSALS_LIMIT in the proposal renegotiation test (753)
* Fix mocks in factories (813)
* Fix instantiation of `OfferProposalFactory` (816)
* Exclude integration test for Windows/Python 3.6 from Github Actions (835)