Flintrock

Changed

* [348], [367]: Bumped default Spark to 3.5.0 and default Hadoop to 3.3.6; dropped support for Python 3.6 and 3.7; added CI builds for Python 3.10, 3.11, and 3.12.
* [361]: Migrated from AdoptOpenJDK, which is deprecated, to Adoptium OpenJDK.
* [362], [366]: Improved Flintrock's ability to cleanup after launch failures.
* [366]: Deprecated `--ec2-spot-request-duration`, which is not needed for one-time spot instances launched using the RunInstances API.
* [369]: Adopted `pyproject.toml` and tweaked Flintrock's Python packaging accordingly. This keeps Flintrock in line with modern Python packaging standards and should be transparent to end-users.

[348]: https://github.com/nchammas/flintrock/pull/348
[361]: https://github.com/nchammas/flintrock/pull/361
[362]: https://github.com/nchammas/flintrock/pull/362
[366]: https://github.com/nchammas/flintrock/pull/366
[367]: https://github.com/nchammas/flintrock/pull/367
[369]: https://github.com/nchammas/flintrock/pull/369

Added

* [296]: Added support for launching clusters into private VPCs. This includes new infrastructure added in [302] to support testing against private VPCs.
* [307]: Added support for Hadoop/HDFS 3.x.
* [315]: Added a new `--ec2-spot-request-duration` option to support setting the EC2 spot request duration.
* [316]: Added a new `--java-version` option and support for Java 11.
* [323]: Flintrock now automatically selects the correct build of Spark to use, based on the version of Hadoop/HDFS that you specify.
* [324]: Flintrock now supports S3 URLs as a download source for Hadoop or Spark. This makes it easy to host your own copies of the Hadoop and Spark release builds in a private bucket.

[296]: https://github.com/nchammas/flintrock/pull/296
[302]: https://github.com/nchammas/flintrock/pull/302
[307]: https://github.com/nchammas/flintrock/pull/307
[315]: https://github.com/nchammas/flintrock/pull/315
[316]: https://github.com/nchammas/flintrock/pull/316
[323]: https://github.com/nchammas/flintrock/pull/323
[324]: https://github.com/nchammas/flintrock/pull/324

Changed

* [285]: Flintrock now configures cluster nodes to use private IP addresses for internal communication. This should improve the reliability of cluster launches and restarts.
* [304]: Fixed a bug in how `UserData` scripts are submitted to new cluster slaves.
* [311]: Changed how Flintrock manages its own security groups to reduce the likelihood of hitting any limits on the number of rules per security group.
* [326]: Switched some internals from using host names to IP addresses, which should improve Flintrock's behavior when running from an EC2 host.
* [329]: Dropped support for Python 3.5 and added automated testing for Python 3.8 and 3.9.
* [334]: Flintrock now ensures that `python3` is available on launched clusters and sets that as the default Python that PySpark will use.

[285]: https://github.com/nchammas/flintrock/pull/285
[304]: https://github.com/nchammas/flintrock/pull/304
[311]: https://github.com/nchammas/flintrock/pull/311
[326]: https://github.com/nchammas/flintrock/pull/326
[329]: https://github.com/nchammas/flintrock/pull/329
[334]: https://github.com/nchammas/flintrock/pull/334

Changed

* [297]: Dropped support for Python 3.4.
* [252]: Flintrock now pins all its transitive dependencies via the files under `requirements/`. This is useful for users who want to build Flintrock themselves.

[297]: https://github.com/nchammas/flintrock/pull/297
[252]: https://github.com/nchammas/flintrock/pull/252

Changed

* [258], [268]: Fixed up support for Python 3.7.
* [264]: Fixed a logging error in `flintrock describe --master-hostname-only`.
* [277]: Fixed a bug in resolving client IP addresses from behind proxy.

[258]: https://github.com/nchammas/flintrock/pull/258
[264]: https://github.com/nchammas/flintrock/pull/264
[268]: https://github.com/nchammas/flintrock/pull/268
[277]: https://github.com/nchammas/flintrock/pull/277

Added

* [242]: Flintrock is now available on Homebrew:

brew install flintrock

This is a community-supported distribution.

[242]: https://github.com/nchammas/flintrock/pull/242

Changed

* [224]: Fixed a problem with some Flintrock config combinations
related to Hadoop.
* [232]: When you destroy a cluster, Flintrock now waits until the
instances are completely terminated before returning.
* [234]: Flintrock now tries more times by default to connect via
SSH, which should provide more launch stability in certain
environments.
* [246]: Fixed some bugs with `flintrock describe` that are exposed
when a cluster is transitioning states (e.g. from running to
terminated).
* [249]: **Flintrock now downloads both Spark and Hadoop from Apache
mirrors by default.** This is a significant change. You can read the
background on what prompted this change in [238].
* [254]: Flintrock no longer configures hadoop-aws automatically due
to version incompatibilities that are difficult to resolve
automatically. Instead, the README now provides additional guidance
on using `s3a://`.
* [259]: Flintrock now correctly ignores tiny devices that show up
on some instance types, like the M5 series on EC2. This fixes the
problems Flintrock had getting HDFS to work on those instance
types.

[224]: https://github.com/nchammas/flintrock/pull/224
[232]: https://github.com/nchammas/flintrock/pull/232
[234]: https://github.com/nchammas/flintrock/pull/234
[238]: https://github.com/nchammas/flintrock/pull/238
[246]: https://github.com/nchammas/flintrock/pull/246
[249]: https://github.com/nchammas/flintrock/pull/249
[254]: https://github.com/nchammas/flintrock/pull/254
[259]: https://github.com/nchammas/flintrock/pull/259

Added

* [178]: You can now see additional output during launch and other
operations with the new `--debug` option.
* [185]: Added a new mount point under `/media/tmp` that can be used
when `/tmp` is not big enough.
* [186]: You can now tag your clusters with arbitrary tags on launch
using the new `--ec2-tag` option. (Remember: As with all options,
you can also set this via `flintrock configure`.)
* [191]: You can now specify the size of the root EBS volume with the
new `--ec2-min-root-ebs-size-gb` option.
* [181]: You can now set the number of executors per worker with
`--spark-executor-instances`.

[178]: https://github.com/nchammas/flintrock/pull/178
[185]: https://github.com/nchammas/flintrock/pull/185
[186]: https://github.com/nchammas/flintrock/pull/186
[191]: https://github.com/nchammas/flintrock/pull/191
[181]: https://github.com/nchammas/flintrock/pull/181

Changed

* [195]: After launching a new cluster, Flintrock now shows the
master address and login command.
* [196], [197]: Fixed some bugs that were preventing Flintrock from
launching Spark clusters at a specific commit.
* [204]: Flintrock now automatically retries starting the Spark and
HDFS masters if it encounters common issues with bringing the
cluster up. This greatly improves launch and restart reliability.
* [208]: Flintrock now provides a hint with possible causes for
certain SSH errors.

[195]: https://github.com/nchammas/flintrock/pull/195
[196]: https://github.com/nchammas/flintrock/pull/196
[197]: https://github.com/nchammas/flintrock/pull/197
[204]: https://github.com/nchammas/flintrock/pull/204
[208]: https://github.com/nchammas/flintrock/pull/208