Tomodachi

**Send messages directly to SQS queues (SQS.SendMessage)**

- Added the function `tomodachi.sqs_send_message` (`tomodachi.aws_sns_sqs_send_message`) to send a message directly to an SQS queue. It mostly works the same way as the `tomodachi.sns_publish` (`tomodachi.aws_sns_sqs_publish`) function, but sends a message directly to the specified SQS queue.

This is useful when you know the recipient queue of a message, for example when manually sending messages to a dead-letter queue or to do command-like messaging instead of event-based messaging.

- The message body of `tomodachi.sqs_send_message` calls is by default built using the `tomodachi.transport.aws_sns_sqs.MessageBodyFormatter` implementation, which will encode the message body as JSON and embed it within the `"Message"` property also using the service specified message envelope functionality.

The message includes some additional properties, mimicing how SNS notifications are represented when they are sent to subscribed SQS queues (with the exception that these messages are of type `"Message"` to differentiate from SNS notifications that are by AWS typed `"Notification"`.

For custom or more advanced setups requiring more flexibility, the implementation can be overridden by providing a callable to the `message_body_formatter` keyword argument of `sqs_send_message` calls. The callable receives a message context object and returns the raw message (as a string) that will be sent to SQS as the message body.

Setting `message_body_formatter` to `None` will disable default message body formatting (and incidentally also the message envelope builder), causing the message body to be as is specified within the `data` argument to the `sqs_send_message` call.

- Trying to send messages to an SQS queue that does not exist will result in a `tomodachi.transport.aws_sns_sqs.QueueDoesNotExistError` exception being raised, which can be caught also by `tomodachi.transport.aws_sns_sqs.AWSSNSSQSException`, `botocore.exceptions.ClientError` or `<botocore_sqs_client>.exceptions.QueueDoesNotExist`.

**Message attributes' improvements and fixes**

- Receives and propagates message attributes for all messages received on a queue, which means message attribute propagation will work for messages published to SNS, using either raw message delivery or not, and/or messages that were directly sent to SQS via SQS.SendMessage calls.

Currently, messages on SQS queues consumed and parsed by tomodachi services still need to be JSON formatted and be embedded within the `"Message"` property (which is how SNS notification messages are represented by default, unless a queue is setup for raw message delivery).

- Fixes a bug that was causing message attributes in binary form to be invalidly processed.

**OTEL updates**

- OTEL auto instrumentation for the new `tomodachi.sqs_send_message` functionality.
- OTEL attributes for AWS SNS+SQS messaging have been updated according to the latest OTEL semantic conventions, where the original publish destination is recorded as `messaging.destination_publish.name`.
- OTEL metrics for AWS SNS+SQS messaging uses the new metric instrument name `messaging.aws_sqs.duration`, which was previously named `messaging.amazonsqs.duration`.

**Type hints and input validation**

- Additional support for the `types-aiobotocore-*` packages to provide improved type hints internally.
- AWS credentials provided to the aiobotocore connector are using stricter type hint annotations.
- Stricter argument validation when requesting aiobotocore connector clients to prevent issues caused by a mistyped AWS service, protecting against accidental use of wrong client for a specific service.

**Handler arguments**

- Added `message_type` (`str | None`) to the list of keyword argument provided transport values that can be used in function signatures for AWS SNS+SQS handlers. Holds the value `"Notification"` for messages received as part of an SNS notification, unless the queue uses raw message delivery. In other cases the value is set to the value from a message body's JSON property `"Type"` if it exists.
- Tailored for more advanced workflows, where more flexibility is needed, `raw_message_body` (`str`) has also been added to the list of keyword argument provided transport values for AWS SNS+SQS handlers. The `raw_message_body` value is set to the full content (non-decoded, as a string) from a received message' `"Body"`, which can be used to implement custom listener with greater access to the raw message data.
- Fixes an issue where the `topic` argument to message handlers in functions was always populated with the value of the `topic` argument to the `tomodachi.aws_sns_sqs` decorator, even if the message was sent to a different topic. The `topic` argument to message handlers in functions will now be populated with the actual topic name destination to where the message was published, if published via SNS. If the message was not published to a topic, the value is set to empty string.

**Other**

- Fixes issue where different OTEL tracers for the same `TracerProvider` could end up with different resource attributes, iff a service class' `name` attribute's value were to be used as the `service.name` OTEL resource attribute (instead set to `"unknown_service"` for tracers that were created before the service initialized).
- Multi-service execution won't accurately distinguish the service name of OTEL tracers, OTEL meters and OTEL loggers. In the rare case where there's multiple `tomodachi` services started within the same Python process, the name from the *first* instrumented service class will now be used as the `service.name` OTEL resource attribute for *all* tracers derived from the same tracer provider. The recommended solution if this is an issue, is to split the services into separate processes instead.
- Refactoring of `tomodachi.aws_sns_sqs.get_queue_url` and `tomodachi.aws_sns_sqs.get_queue_url_from_arn` with better support for additional types of input and/or potentially prefixing of queues. The function also now caches the queue URL to avoid unnecessary calls to the AWS API.
- Support for `aiobotocore` 2.10.x releases and 2.11.x releases.
- Documented the lifecycle termination process for services, for example when they receive a `SIGINT` or `SIGTERM` signal. Added details describing how handlers are awaited and for how long as part of the graceful termination sequence.
- Converted repo documentation from RST format to Markdown. It finally happened.

- Fixes an issue with the custom backport class of `ConsoleRenderer` extending `structlog.dev.ConsoleRenderer` (which caused an exception to be raised upon service start, when using `structlog` 23.3.0 or 24.1.0).
- Uses the native `ConsoleRenderer` class from `structlog` when a modern version of `structlog` is used (23.3.0 or newer).

- Updated version constraints of the `opentelemetry-exporter-prometheus` dependency to use the correct (non-yanked) version range.
- Support for `aiobotocore` 2.8.x releases and 2.9.x releases.

- Support for Python 3.12. Python 3.12 has been added to test matrix and trove classifiers.
- Supports `aiohttp` 3.9.x versions.
- Updated OpenTelemetry distro to handle MetricReaders and MetricExporters in a more flexible way (similar to how it was changed to be done in `opentelemetry-sdk` 1.21.0).
- Log entries from `tomodachi.logging` on OTEL instrumented services will now include TraceContext (`span_id`, `trace_id` and `parent_span_id`) on non-recording (but valid) spans.

- Uses less strict version constraints on OpenTelemetry dependencies to make it easier to use newer versions of OpenTelemetry libraries together with `tomodachi` services.
- Passes FIFO message attributes (`message_deduplication_id` and `message_group_id`) to `tomodachi` handler functions as keyword argument provided transport values. (github: **filipsnastins**)

**New and updated**

- Major refactoring of how logging is done, introducing `tomodachi.get_logger()` built on `structlog`. Contextual logger available for all handlers, etc.
- Provides the option of hooking in `tomodachi` to a custom logger the user provides.
- Adds instrumentation for `opentelemetry` (OTEL / OpenTelemetry) that can be enabled if `tomodachi` is installed using the `opentelemetry` extras.
- OTEL auto instrumentation can be achieved by starting services using either the `tomodachi run` argument `--opentelemetry-instrument` (equivalent to setting env: `TOMODACHI_OPENTELEMETRY_INSTRUMENT=1`) or using the `opentelemetry-instrument` CLI.
- An experimental meter provider for exporting OpenTelemetry metrics on a Prometheus server can be used by installing the `opentelemetry-exporter-prometheus` extras and using the `OTEL_PYTHON_METER_PROVIDER=tomodachi_prometheus` environment value in combination with OTEL instrumentation.
- Adds the option to enable exemplars in the Prometheus client for OpenTelemetry metrics, to be able to link to traces from collected metrics in Grafana, et al.
- The HTTP body for requests with a body is read before the handler is called and if the connection was closed prematurely before the body could be read, the request will be ignored.
- Replaces the banner shown when starting a service without `--production`. The banner now includes the operating system, architecture, which Python runtime from which virtualenv is used, etc. in order to aid debugging during development for issues caused by environment misconfiguration.
- Updated the CLI usage output from `--help`.
- Added a value `tomodachi.__build_time__` which includes the timestamp when the build for the installed release was done. The time that has passed since build time will be included in the start banner.
- Makes use of `asyncio` tasks instead of simply awaiting the coroutines so that the context from contextvars will be propagated correctly and not risk being corrupted by handlers.
- Added an internal lazy meta importer to ease renaming and deprecations of modules.
- Added additional lazy loading of submodules.
- Each argument for `tomodachi run` is now accompanied with an environment variable to do the same. For example `--log-level warning` can be achieved by setting `TOMODACHI_LOG_LEVEL=warning`.
- Updated documentation with additional information.

**Potentially breaking changes**

- The complete refactoring of logging changes how log entries are being emitted, both in the way that the event / message of the log records has changed, but also how a log handler is now also added to the `logging.root` logger on service start.
- Third party log records will if propagation is enabled also be processed in `tomodachi.logging` which may cause duplicate log output depending on how the third party logger is configured.
- Removed the `log_setup()` function that previously was added to the service object on class initialization and that was used to setup log output to disk.
- Tracebacks for uncaught exceptions are now extracted to only include frames relevant to the service application and not the internal `tomodachi` frames, which usually will be uninteresting for debugging.

**Bug fixes**

- Fixes exception catching of lifecycle handlers (`_start_service`, `_started_service`, etc.) which previously could stall a service raising an exception while starting, instead of exiting with a non-zero exit code.
- Bug fix for an issue which could cause the watcher to fail to restart the service after a syntax error was encountered.
- Adds some missing type hint annotations.
- Added additional logging of uncaught exceptions that previously may have been silenced.
- Fixed that the `--log-level` CLI argument value is actually applied to loggers.
- Fix for a race condition which could freeze a process if a service was manually stopped (interrupted with ctrl+c) before it had called its first lifecycle function (`_start_service`).

**Deprecations**

- Added deprecation warnings for more legacy functionality to give notice that those functions will be removed in a future release.
- The use of the `log()` function added to the service object is deprecated. Use the `structlog` logger from `tomodachi.get_logger()` instead.
- Using the `RequestHandler.get_request_ip` is deprecated. Instead use the `tomodachi.get_forwarded_remote_ip()` function.
- Deprecated the use of the CLI argument `-c` (`--config`) which could be used to set object attributes from a JSON file. A better pattern is to read optional config data from an environment variable.