Localstack

Change Log

1. New Features

* add initial ASF (AWS Service Framework) SQS provider implementation
* add automatic Docker network detection for Lambda containers
* add support for dynamic SSM/SecretsManager references in CloudFormation templates
* add version info to internal health HTTP endpoint

2. Enhancements

* add support for gzipped responses in SSL proxy via Accept-Encoding headers
* lowercase 'accept' headers in API Gateway for parity with AWS
* add negative test for SNS FIFO topic validation, migrate SNS tests from unittest to pytest
* add ability to customize default IAM/STS user via TEST_IAM_USER_ID/TEST_IAM_USER_NAME
* add integration test that passes binary data via APIGateway to Lambda
* refactor StepFunctions multi-region support to use upstream changes
* add proper validations for SNS/SQS integration with fifo queues
* allow provider overrides to be passed to the started LocalStack container
* ensure SequenceNumber is present in Lambda events from DynamoDB Streams
* support updating of API Gateway resources via PATCH operations
* refactor logging code to replace string interpolation with passing arguments to log methods
* refactor code to remove unnecessary list comprehensions
* refactor code to remove mutable default arguments
* pull out subtypes instance manager into separate util class for reusability
* update and clean up outdated documentation
* replace dict and list function calls to literal syntax
* remove `uname` command in system check for windows compatibility

3. Bug Fixes

* fix API gateway proxy resources
* fix passing of request parameters to API Gateway HTTP integrations
* fix Firehose-ElasticSearch integration, allow S3Backup AllDocuments with ElasticSearchDestination
* fix multiple service container creation
* fix extraction of filter values for describing EC2 prefix lists
* fix single-space env default in docker-compose
* fix RenovateBot config to only enable patch updates

Announcements

* **Security fixes**: This release upgrades **log4j** dependencies to version `2.17.0` to fix the critical security vulnerabilities [CVE-2021-44228](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228), [CVE-2021-45046](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046)

Change Log

1. New Features

* add initial support for region-based in-place partition rewriting
* add "--offline" pytest flag, skip "online-only" tests, fix some tests
* add SKIP_SSL_CERT_DOWNLOAD option to allow skipping download of SSL cert

2. Enhancements

* bump version of Java utils to 0.2.18 to fix log4j vulnerability CVE-2021-44228, CVE-2021-45046
* refactor README.md to make it crisp and readable
* refactor Lambda logic to remove local executor Callable from model entities
* automatically pull non-cached Docker images on image inspection
* replace dict calls with comprehensions
* small fixes to allow running LocalStack with podman
* reduce log level of edge port configuration hook
* correctly wait for stay-open port to be available, fix fallback to exec mode
* add compatibility checks for S3 copy object with metadata
* add proper error response message to reject empty SNS messages
* minor refactoring to use negative index -1 to get the last element of sequence
* minor refactoring of Lambda API for better extensibility

3. Bug Fixes

* fix updating of HTTP method in message handler chain to avoid None methods
* fix misc. tests failing for non-default region
* fix Terraform test issue related to SQS changes in the AWS provider
* fix association of VPCs in Route53 HostedZone responses
* fix CloudFormation updates for EC2::Instance with empty SecurityGroups property
* fix listing of KMS signing keys for asymmetric key pairs

Announcements

* Upcoming breaking change: This version introduces the `LEGACY_PERSISTENCE` config, which determines whether the legacy persistence mechanism (based on API calls record&replay) should be used. Currently still defaults to `LEGACY_PERSISTENCE=1`, but the logic will be disabled by default and may get removed entirely in a future release. (Please note: this is only affecting the legacy persistence in Community - if you're using the Pro version, you can ignore this message.)

Change Log

1. New Features

* add LocalStack Package Manager (lpm) CLI
* add plugin-based hooks for bootstrapping and infra startup
* add functionality for custom Elasticsearch backends
* add support for stay-open mode for docker-reuse Lambda executor
* add cli command to show current config
* add initial integration of Route53Resolver API
* add multi-region support for StepFunctions state machines

2. Enhancements

* refactor handler chain and add MessageModifyingProxyListener to enable modification of requests/responses
* add Directories config object and introduce directory structure
* use unittest.mock.patch.object for overriding config params
* publish logs per metric and remove faulty metric filter behavior in CloudWatch API
* add Docker tags for minor and patch versions
* exclude Elasticsearch >=7.14.0 client versions for OpenSearch compatibility
* add SIGINT signal handler for CLI to enable clean shutdown across different operating systems
* clean up obsolete patches for event_rules in EventBridge API
* slightly refactor StepFunctions install logic for better extensibility
* add flag to check port availability in start_proxy_server(..) to fix test flakes
* increase timeout when waiting for edge port to become available
* allow redirection of var libs to static libs inside the container / move dependencies to correct folders
* update startup hook that initializes and restores persistence
* download test-jar for Lambda integration tests with `make init-testlibs`
* filter warnings about "tail unrecognized file system" from Lambda logs
* add basic Architectures support to Lambda API
* get hostname by using gethostname() instead of reading /etc/hostname
* add patch utility
* add .localstack config directory and config profiles
* add initial support for KMS Sign API
* skip time expiration validation for presigned S3 URL when `S3_SKIP_SIGNATURE_VALIDATION=True`
* remove dead event logging code
* move machine id cache to new cache directory
* add missing attributes for Lambda::Function CloudFormation resources
* add "is running" check before restarting Kinesis and StepFunctions services
* add module init file for localstack.runtime
* add support for SNS delivery logs stored to CloudWatch Logs
* explicitly handle S3 OPTIONS request with "Access-Control-Request-Method" header
* add psutil as CLI requirement
* strip extra xmlns attributes in PutBucketNotification responses for AWS Rust SDK compatibility
* update Makefile to use new cli detach flag for smoke test
* patch CFN stack outputs for API Gateway
* make can_use_sudo use non-interactive mode
* migrate several tests from unittest -> pytest
* re-arrange Dockerfile commands to decrease image size and increase number of cache hits
* pin Docker base images, enable DependaBot/RenovateBot to update once a week
* apply boolean lowercase conversion in IAM responses for all clients (not only Node.js SDK)
* enable Docker buildkit cache inlining, use remote Docker layer cache
* patch `LogStream.filter_log_events` to use `get_pattern_matcher`
* minor refactoring of EventBridge utils, fix location of `EVENTS_TMP_DIR`
* minor: improve parameter checks and error responses for SES SendEmail

3. Bug Fixes

* fix returning formatted date string for requestTime in Lambda API GW events
* fix edge routing for API Gateway invocation URLs when Signature= query param is passed
* fix SNS pagination to support large CloudFormation stacks with very large number of topics
* fix lpm to return non-zero exit code if package installation fails
* fix using custom Docker images for nodejs14.x/python3.9 only if no custom registry is provided
* fix MD5 check on S3 requests with "chunk-signature="
* fix "localhost" region in requests headers for compatibility with NoSQL Workbench
* fix Docker flags parsing in configure_container
* fix tests and remove obsolete patch for CloudWatch metrics filtering
* fix setting HMAC/non-HMAC credentials when injecting internal Authorization headers in requests
* fix OldImage/NewImage in DynamoDB->Kinesis event payloads on updating/deletion of items
* fix creation of multi route table associations in CFN EC2::SubnetRouteTableAssociation resources
* fix deleting objects from non-existing S3 buckets
* fix text/xml content-type header in STS responses
* fix install_go_lambda_runtime for multi-arch build
* fix forwarding of unprintable chars to SQS DLQ
* upgrade pyopenssl version to fix OpenSSL issue, refactor SSL cert generation
* minor fix checking for dict/CaseInsensitiveDict in merge_recursive(..) util function

Major Changes

This release introduces three major changes

* A Debian-based multi-platform Docker image of LocalStack, with full support for amd64 and experimental support for aarch64 (see 4921 for known limitations and progress)
* Upgrade to Python 3.8
* Lazy-loading of services as the default behavior

Debian-based multi-platform (amd64, arm64) docker image

4754 introduced a multi-platform docker build to support both amd64 and arm64. Alpine and musl made things unnecessarily difficult, so we switched to a Debian buster base image. Post-init (e.g., in `/docker-entrypoint-initaws.d`) scripts that expect an Alpine environment (e.g., use `apk` commands to install things into the LocalStack container) will stop working.

Lazy-loading of services

As part of an ongoing effort to improve startup performance, we have introduced a new code loading and service startup mechanism to allow lazy-loading of services. After LocalStack starts, services used to be in the state `running`, indicating that they are loaded and ready to serve requests. To allow more fine-grained state handling for services with lazy loading, we have introduced a new type of state called `available`.
When starting LocalStack, services are by default in the state `available`, which means that clients can start making requests to the services, and at the first time the service is hit, all the necessary code is loaded and the backend service is started, at which point the service after which the service will be in the `running` state. The environment variable `EAGER_SERVICE_LOADING` controls this behavior.

Scripts that depend on the output of `/health` being `running` will require setting setting `EAGER_SERVICE_LOADING=1`, which will start all services eagerly. This works together with the `SERVICES` variable to create the behavior from localstack <= 0.13.0.

(An alternative solution is to update your startup logic to accept either `running` or `available` as the healthy service state - e.g., see details in [this issue](https://github.com/localstack/localstack/issues/4904)).

Change log

1. New Features

* make lazy service loading the default behavior
* debian-based multi-platform support
- multi-platform support in localstack codebase
- finalize multi-platform build
- initial version of multi-platform-build
* support multiple concurrent elasticsearch clusters:
- implement one cluster per domain-support for elasticsearch
- implement custom endpoint routing for elasticsearch clusters
* add simple UI for deploying CloudFormation stacks from public template URLs
* add delete support for AWS::SSM::Parameter CloudFormation resources
* add initial support for Firehose processors to transform records written to targets
* add ability to short circuit wait/retries

2. Enhancements

* use serving.Server abstraction for ElasticMQ server to resolve startup issues
* add Werkzeug>=2.0 as requirement
* use werkzeug Request object in asf
* use werkzeug Response object in asf
* invalidate pro test cache for changes in both
* bump moto version to 2.2.5
* minor: correct typos in README
* add support for {"exists": false} in SNS message filtering
* add support for CFN resource AWS::Logs::LogStream
* minor: adjust API GW logic and tests for latest upstream moto changes
* convert boolean strings to lowercase in IAM responses for JS SDK compatibility
* add LEGACY_DOCKER_CLIENT and EAGER_SERVICE_LOADING to CONFIG_ENV_VARS
* minor: add property to API GW resource context class to extract identity information
* refactor logic for pulling specific Docker image tags, add more tests
* implement generic approach for routing internal REST resources
* extend logic and add more tests for escapeJavaScript(..) velocity template function
* remove custom parameter by label filtering, leverag logic from upstream
* upgrade moto-ext dependency to latest version
* Format JSON stack template for better readability in UI
* minor: remove print(..) statement
* support ranges of host ports to be mapped to a container port
* add missing attributes in Firehose records sent to destinations

3. Bug Fixes

* minor fix to deal with missing Tags attribute in EC2::RouteTable
* fix filtering by tags when fetching state of EC2::RouteTable CloudFormation resources
* fix URL encoding to properly handle '(none)' base paths in API Gateway
* fix arn parsing in aws_stack
* minor: fix API GW test assertion after recent moto upstream changes (part 2)
* minor: fix API GW test assertion after recent moto upstream changes
* minor fixes in Lambda and API Gateway logging / error handling
* fix extracting attributes from Firehose records
* revert moto to version 2.2.4 to fix builds temporarily

**1. New Features**

* major: implement lazy-loading of services
* introduce ApiInvocationContext class to encapsulate API Gateway request context information
* add AWS Service Framework parsers, serializers, and tests
* add AWS Service Framework API and scaffolding CLI
* add initial lifecycle hook abstraction for services
* add `OUTBOUND_HTTP_PROXY` option for proxies when downloading external resources
* add temporary support for python3.9 runtimes via mlupin/docker-lambda images
* add support for more Elasticsearch versions

**2. Enhancements**

* add object-oriented moto server abstraction
* update moto patch to fix CDATA wrapper for SQS message attribute values
* add option to set permissions for save_file(..), save local config file with 0o600 permissions
* upgrade Terraform version to fix CI builds
* refactor API GW logic for easier extraction of resource/method details from invocation context
* add test-invoke-method to test API Gateway endpoints
* add requestId attribute to Lambda event from API Gateway integration
* add more SQS tests to prepare for new SQS implementation
* refactor DynamoDB to use serving.Server
* update Elasticsearch default version in es_api.py
* support version qualifiers for managing Lambda policy statements
* correctly format Lambda function inline code on CloudFormation update
* refactor serving.Server to fix ElasticsearchCluster
* add headers from S3 object and correct codes to S3 website responses
* add defaults for SSM parameter name in CloudFormation model
* extend create_zip_file(..) to support 'append' mode and specifying root directory
* update default/cached Elasticsearch version to 7.10
* extract correct region from target ARN for S3 notifications
* add ability to skip content gzipping for individual services responses
* make boto client creation in aws_stack.connect_to_service thread safe
* add SSM->EventBridge integration to send notifications about parameter changes
* add concurrency control for multiserver startup
* add multi-region support for EventSourceListenerSQS
* add setup.cfg and update build configuration
* pin version of "pip" to fix setup step in CI
* add support for "::" to specify a custom Java Lambda handler method in local executor mode
* fix health endpoint to support feature states
* refactor extract_region_from_headers to make it more reusable
* add generic fallback health check for local APIs
* adjust exposed ports in docker-compose.yml
* add logic and tests to gracefully handle zipping of empty directories
* use Python time for do_run cache time comparison
* add support for routing events to log groups, fix event pattern matching/mapping for prefix filter
* prefix Lambda container names with MAIN_CONTAINER_NAME to support multiple instances in parallel
* allow partial SSE specification and auto-create managed KMS keys for DynamoDB tables
* update troubleshooting guide to not set DOCKER_HOST when using Lambda remote mode in CI
* minor: use existing util function to extract region from ARN
* minor: add generate_default_name_without_stack utility method

**3. Bug Fixes**

* fix docker_utils import in localstack status cli command
* fix AWS::SecretsManager::Secret CloudFormation ARN handling and tests
* fix restarting of DynamoDB process
* fix local execution of Java Lambdas if handler method is implementing `RequestHandler` interface
* fix issues in AWS service framework dispatching
* fix error response when attempting to delete non-existing Lambda
* fix scaffolding for remaining AWS services
* fix Elasticsearch plugin download when building docker base image
* fix check for None values in response multi_value_headers
* fix overwrite behavior for PUT requests to the health endpoint
* fix updates for AWS::IAM::Role in CloudFormation
* fix state checking for active service provider
* fix statement/policy handling for Events::EventBusPolicy CFN resource
* upgrade version of Terraform in base image to fix recent CI build issues

**1. New Features**

* add initial support for KMS asymmetric keys
* add initial support for KMS grant operations
* add support for and/or/condition CFN intrinsic functions
* add support for states:::events:putEvents to publish events from StepFunction
* add wildcard support for `EXTRA_CORS_ALLOWED_ORIGINS` configuration

**2. Enhancements**

* move attribute defaults from template_deployer to service models
* cache single instance of request context to enable mutability
* refactor Lambda API delete_function logic for better extensibility
* pass ENABLE_CONFIG_UPDATES environment variable to Docker container from CLI
* support case-sensitive response headers, add ETag to CORS exposed headers for S3 JS SDK compatibility
* add pagination to get_lambda_log_events(..) util function to fix Lambda tests
* log emails sent via SES send_raw_email into local log file
* add ability to pass invocation handler to test HTTP server for API GW integration tests
* add missing event attributes and increase timeout in Java Lambda tests
* rename docker->docker_utils to avoid import/aliasing issues
* minor refactoring of inter-service message forwarding for better extensibility
* add simple util classes ObjectIdHashComparator and ArbitraryAccessObj
* add helper function to mock a request context with a particular region
* refactor and remove fix_resource_props_for_sdk_deployment(..) in CFN deployer
* move delete patches in template deployer to CFN service models
* refactor error handling and add MOCK_UNIMPLEMENTED config to gracefully handle unimplemented APIs
* add ability to pass --workdir to launched containers in Docker client
* allow Lambda plugins to return invocation results directly, create plugin to handle LAMBDA_FORWARD_URL
* increase ElasticSearch cluster startup timeout
* skip forwarding Lambda invocation result if on_successful_invocation is None
* add missing edge routing logic for AWS SDK v2
* add support for s3:ObjectCreated:Copy event in the presence of X-Amz-Copy-Source headers
* add Docker client methods to inspect networks
* wait until port is closed on DynamoDB shutdown/restart
* pass CORS configuration options and Thundra Node.js/Python agent config from CLI to the container
* add util function to restart StepFunctions service process
* minor: check for empty files in download_and_extract(..) util function
* implement PluginDisabled exception to handle disabled plugins
* refactor DDBStreams/ES/Firehose APIs to use RegionBackend classes

**3. Bug Fixes**

* fix potential race condition in Docker container start/wait logic
* fix CFN SecretsManager::Secret to properly return secret ARN as Ref
* fix CFN resource type lookup
* fix deploying Lambdas with colliding names due to common prefixes
* fix issue with single quotes in payload for Node.js Lambda in local execution mode
* fix deployment of DynamoDB table with SSE disabled in Terraform definition
* fix selection of sub-elements in SQS listener XML responses
* fix Lambda SNS event source attribute as `aws:sns` instead of `localstack:sns`
* minor fix to avoid running multiple edge proxies on the same port