Baseplate

Important News

This will be the last version of Baseplate.py to support Python 3.5 and lower (including 2.7). We will continue to publish bugfixes for this release going forward to support services stuck on old Pythons, but all new development will expect Python 3.6 or newer.

New Features

Ratelimit tools

Baseplate now has helpers for maintaining ratelimit counters in Memcached or Redis. You can use this to correctly apply ratelimits to actions in your application.

See [the docs](https://baseplate.readthedocs.io/en/v0.30.0/baseplate/ratelimit.html) for more information.

More runtime metrics

The previous release (v0.29) added a new system of per-process server metrics. This release adds two more things to watch: garbage collector stats and event loop blocker monitoring. These metrics will help you understand if your application is stalling in ways that would cause weird p99 spikes across many requests.

See [the docs](https://baseplate.readthedocs.io/en/v0.30.0/cli/serve.html#process-level-metrics) for more information.

Credential secrets

Services often need to securely store username/password pairs. Baseplate now has a convention for doing so called a credential secret. In addition, the sqlalchemy integration now uses this new credential type and you can expect other integrations to do so in the future.

See [the secrets store docs](https://baseplate.readthedocs.io/en/v0.30.0/baseplate/secrets.html#baseplate.secrets.SecretsStore.get_credentials) for more information on credential secrets and [the sqlalchemy integration docs](https://baseplate.readthedocs.io/en/v0.30.0/baseplate/context/sqlalchemy.html#baseplate.context.sqlalchemy.engine_from_config) for how to use that with SQL databases.

Changes

* The [`FileWatcher` can now pass various options](https://baseplate.readthedocs.io/en/v0.30.0/baseplate/file_watcher.html#baseplate.file_watcher.FileWatcher) (encoding, binary mode, etc.) through to the `open` call when it loads the watched file.
* The configuration prefix used by [`secrets_store_from_config`](https://baseplate.readthedocs.io/en/v0.30.0/baseplate/secrets.html#baseplate.secrets.secrets_store_from_config) and [`experiments_client_from_config`](https://baseplate.readthedocs.io/en/v0.30.0/baseplate/experiments/index.html#baseplate.experiments.experiments_client_from_config) is now configurable.
* **Potentially breaking**: an accidental `logger` attribute has [been removed](https://github.com/reddit/baseplate.py/commit/12b6967f09612994bb4200db0255e9ec435e5304) from the context object.
* [Timers can now be sent manually without start/stop.](https://baseplate.readthedocs.io/en/v0.30.0/baseplate/metrics.html#baseplate.metrics.Timer.send)
* When a metrics batch is too large, we also log the counters found in that batch. This error generally indicates that the service is doing far too much in a single request and the counters can help figure out what operation is being repeated many times.
* Experiments can now use more targeting operators like gt, lt, etc. and specify ranges of values they apply to.
* Trace publishers now send larger batches to reduce load on Zipkin.
* `baseplate-tshell` now supports IPython 5+.
* `baseplate-tshell` now activates `readline` for proper text editing.
* Development has moved into the OneVM and the tooling has been modernized.

Bug Fixes

* A race condition in the Cassandra integration was fixed. **You should no longer get "timer already stopped" errors from the Cassandra timers.**
* A regression in parsing the `Sampled` header on upstream spans in Pyramid services was fixed.
* Thrift header names are now case-insensitive. This allows them to transit systems like Envoy that canonicalize the names to lower case.
* The message queue helper properly prints messages without `b""` artifacts.
* An exception in the experiments framework is now caught and turned into a mismatch safely.

-18,5 +18,5 raven==5.27.0
-Thrift==0.1
+thrift==0.12.1


Additionally, the Thrift compiler for Apache Thrift is called `thrift` rather than `thrift1`. If you're using the compiler directly you'll need to update this. Baseplate's built in thriftfile compilation steps handle this automatically.

This new updated compiler has a few differences which your thrift IDL and application code will need to take into account:

* The `float` type in FBThrift isn't available in Apache Thrift, only the larger `double` type is. Unfortunately, this is a breaking change on the wire as the two types have quite different byte representations due to their different sizes. For an actively used field, you can make a new `double`-typed field and have your application populate or read both the float and double fields. Once all clients are using the new field you can drop the old one and then move to the new Baseplate.
* Optional arguments to RPC methods do not get a default `=None` in the generated code anymore. Clients will need to ensure they're passing values for all parameters.
* A list of keywords from various languages (e.g. `next`) is now blacklisted for use in field names in Thrift. If you have any fields with names like this, the new compiler will balk. Thankfully this is a purely code-side change and has no effect on how things look on the wire so you can just update your code without worrying about clients.

Changes

* Accept B3- prefix for tracing headers to allow support for other tracing clients.
* Report crashes that happen outside server spans to Sentry. This includes stuff that happens during request parsing before the application receives the request.
* More additions to experiment framework.
* Targeting overrides
* SimpleExperiment targeting
* Include span information in experiment events

Bug fixes

* Fix creation of context attributes on local spans
* Fix local span support in sqlalchemy context client

New Features
* Support new experiment bucketing logic (186).

Changes
* Add kombu to docs requirements.
* Skip queue_consumer_tests if kombu not present.
* Swallow and log exceptions when experiment config is bad (191).

Bug Fixes

* Fix build on BSD.

New Features

Queue consumer support with Kombu

Baseplate now has first class support for consuming messages from queue brokers like RabbitMQ using [Kombu](http://kombu.readthedocs.io/en/latest/). The full trace and diagnostic framework works here.

python
from kombu import Connection, Exchange
from baseplate import queue_consumer

def process_links(context, msg_body, msg):
print('processing %s' % msg_body)

queue_consumer.consume(
baseplate=make_baseplate(cfg, app_config),
exchange=Exchange('reddit_exchange', 'direct'),
connection=Connection(
hostname='amqp://guest:guestreddit.local:5672',
virtual_host='/',
),
queue_name='process_links_q',
routing_keys=[
'link_created',
'link_deleted',
'link_updated',
],
handler=process_links,
)


[See the documentation for more details](http://baseplate.readthedocs.io/en/latest/baseplate/queue_consumer.html).

Changes

* The memcached instrumentation now adds details about each call to span tags. This includes key names, key counts, and other settings.
* When preparing CQL statements with the Cassandra integration, Baseplate will now cache the prepared statement for you. This means you can call `prepare()` every time safely.
* The secret fetcher daemon can now be run in a single-shot mode where it exits immediately after fetching secrets. This can be used for situations like cron jobs in Kubernetes.
* When installing as a wheel, the baseplate CLI scripts no longer have a Python version suffix. `baseplate-serve2` -> `baseplate-serve`.
* The Zipkin tracing observer can now ship spans to a sidecar span publisher daemon rather than sending from within the application itself.
* There are now new methods to check experiment names are valid and to get lists of all active experiments.
* Experiments now send exposure events.

Bug Fixes

* Fix a case where connection failures in the thrift connection pool implementation would cause the pool to lose connection slots and eventually be depleted.
* Fix an issue where for r2 experiments with low bucketing and 3 total treatments, bucketing is uneven.

New Features

baseplate-tshell

You can now fire up a REPL shell in a Thrift service's context for debugging. This is patterened off of `pshell` from Pyramid and supports IPython's REPL if installed.


$ baseplate-tshell2 example.ini
Python 2.7.6 (default, Nov 23 2017, 15:49:48)
Type "copyright", "credits" or "license" for more information.

IPython 2.2.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.

Available Objects:

app This project's app instance
context The context for this shell instance's span
In [1]: context.redis.ping()
Out[1]: True


Changes

* Multiple increments of the same statsd counter in the same batch are now coalesced together before serialization to reduce metric datagram size. This reduces bandwidth usage for most non-trivial applications since span success/failure counters happen often.
* Exceptions raised from the OS during MessageQueue instantiation now come with hints of how to resolve the issue.
* Event logging in the experiments framework is now extensible and defaults to just logging at DEBUG level.
* Success/failure counters (added in v0.20) are now sent for server spans as well as client spans.
* The `EdgeRequestContext` now has a `service` property for use when the request's authentication token identifies an (internal) service rather than a user as the principal. The object has a single `name` property to get the authenticated service's name. This can be used to whitelist services for specific multi-user access to data.


Bug Fixes

* Fix a crash in the `MetricsBaseplateObserver` when a metrics batch gets too big for IP fragmentation. Just log a warning and swallow the error now.
* Don't put connections back in thrift connection pool after unexpected low-level errors from the remote application. This was a cause for at least some "TTransportException: read 0 bytes" errors since the connection was broken but had been returned to the pool for the next request to fail on.
* Fix baseplate-healthcheck3 against UNIX domain socket endpoints. There was a Python 3 incompatibility.
* Include Makefile in setuptools MANIFEST.

Upgrading

Event logging

Remove the `event_queue` parameter from your `ExperimentsContextFactory` object and add an appropriately constructed `EventLogger` instead.