The Metaflow 2.4.0 release is a minor release and includes a *breaking change*
- [Breaking Changes](v2.4.0_breaking)
- [Change return type of created_at/finished_at in the client](692)
- [Bug Fixes](v2.4.0_bugs)
- [Better error messages in case of a Conda issue](706)
- [Fix error message in Metadata service](690)
- [New Features](v2.4.0_features)
- [S3 retry counts are now configurable](700)
- [New datastore implementation resulting in improved performance](580)
- [S3 datatools performance improvements](697)
<a name="v2.4.0_breaking"></a>Breaking Changes
<a name="692"></a>Change return type of created_at/finished_at in the client (692)
Prior to this release, the return type for `created_at` and `finished_at` properties in the Client API was a timestamp
string. This release changes this to a `datetime` object, as the old behavior is considered an unintentional mis-feature
(see below for details).
How to retain the old behavior
To keep the old behavior, append an explicit string conversion, `.strftime('%Y-%m-%dT%H:%M:%SZ')`, to
the `created_at` and `finshed_at` calls, e.g.
run.created_at.strftime('%Y-%m-%dT%H:%M:%SZ')
Background
The first versions of Metaflow (internal to Netflix) returned a `datetime` object in all calls dealing with timestamps in
the Client API to make it easier to perform operations between timestamps. Unintentionally, the return type was changed
to string in the initial open-source release. This release introduces a number of internal changes, removing all
remaining discrepancies between the legacy version of Metaflow that was used inside Netflix and the open-source version.
The timestamp change is the only change affecting the user-facing API. While Metaflow continues to make a strong promise
of backwards compatibility of user-facing features and APIs, the benefits of one-time unification outweigh the cost of this
relatively minor breaking change.
<a name="v2.4.0_bugs"></a> Bug Fixes
<a name="706"></a>Better error messages in case of a Conda issue (706)
Conda errors printed to `stderr` were not surfaced to the user; this release addresses this issue.
<a name="690"></a>Fix error message in Metadata service (690)
The code responsible for printing error messages from the metadata service had a problem that could cause it to be unable to print the correct error message and would instead raise another error that obfuscated the initial error. This release addresses this issue and errors from the metadata service are now properly printed.
<a name="v2.4.0_features"></a>New Features
<a name="700"></a>S3 retry counts are now configurable (700)
This release allows you to set the number of times S3 access are retried (the default is 7). The relevant environment variable is: `METAFLOW_S3_RETRY_COUNT`.
<a name="580"></a>New datastore implementation resulting in improved performance (580)
The datastore implementation was reworked to make it easier to extend in the future. It also now uploads artifacts in parallel to S3 (as opposed to sequentially) which can lead to better performance. The changes also contribute to a notable improvement in the speed of `resume` which can now start resuming a flow twice as fast as before. Documentation can be found [here](https://github.com/Netflix/metaflow/blob/master/docs/datastore.md).
<a name="697"></a>S3 datatools performance improvements (697)
The S3 datatools better handles small versus large files by using the `download_file` command for larger files and using `get_object` for smaller files to minimize the number of calls made to S3.