Hello folks! 👋
This is the first version of the Athena Federation Python SDK that is ready for real-world use cases.
Please note! This is still a pre-release, so if you run into any issues or questions, please ask first in the [Discussions](https://github.com/dacort/athena-federation-python-sdk/discussions).
Overview
This release introduces support for Splits and automatic Spill-to-S3 functionality.
- Splits can be used to parallelize queries to your Lambda function. Any properties you return in the `splits` method will be passed to the `record` method as a `split` parameter. You can then use your properties to filter the data source being queried.
- Spill-to-S3 is used in the event that the size of the data being returned exceeds 6MB, the maximum response for a Lambda function. This is automatically handled in the `ReadRecordsRequest` portion of the `AthenaLambdaHandler`.
Usage
An example data source is available in the [`example/`](https://github.com/dacort/athena-federation-python-sdk/tree/main/example) subdirectory. It hard-codes sample values, but illustrates how to create a new data source as well as connect it to a Lambda handler in Python.
Instructions on creating a Lambda function can be found [in the README](https://github.com/dacort/athena-federation-python-sdk#creating-your-lambda-function).
Happy coding! 🧑💻