This release is a significant, backwards incompatible change with respect to coffea 0.7 analysis code and patterns. We have made sure the translation from coffea 0.7 to this version of coffea is not unendurably difficult.
The coffea 202x series is a major upgrade to coffea that incorporates uproot5, awkward2, dask-awkward, dask-histogram, and fsspec. It uses all of these packages to create a cohesive, scalable, and fluidly evolving analysis and analysis-framework design platform that is convenient, well performing, and robust. Using this release, analysis design and execution is made significantly more smooth and manageable at all stages from initial ideation and data exploration to producing publication quality results.
It includes significant new features like: failed file handling and partial job completion, skimming, high-level dataset handling, streamlined interfaces to machine learning inference at scale, access to columnar jet-clustering, ... .
Many thanks to everyone who made this release possible including: jpivarski agoose77 nsmith- douglasdavis yimuchen iasonkrom valsdav
What's Changed
* build(deps): bump pypa/gh-action-pypi-publish from 1.7.1 to 1.8.1 by dependabot in https://github.com/CoffeaTeam/coffea/pull/774
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/781
* build(deps): bump pypa/gh-action-pypi-publish from 1.8.1 to 1.8.4 by dependabot in https://github.com/CoffeaTeam/coffea/pull/780
* Awkward v2 transition by lgray in https://github.com/CoffeaTeam/coffea/pull/736
* build(deps): bump pypa/gh-action-pypi-publish from 1.8.4 to 1.8.5 by dependabot in https://github.com/CoffeaTeam/coffea/pull/782
* Get tests in passing state, mark heisenbugs by lgray in https://github.com/CoffeaTeam/coffea/pull/789
* update awkward2 pin by lgray in https://github.com/CoffeaTeam/coffea/pull/785
* Use codecov action now by lgray in https://github.com/CoffeaTeam/coffea/pull/791
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/787
* update awkward, dask_awkward, dask_histogram pins by lgray in https://github.com/CoffeaTeam/coffea/pull/792
* Use uproot chunking for uproot.dask, repin uproot by lgray in https://github.com/CoffeaTeam/coffea/pull/793
* Pin dask < 2023.4.0 while dask_awkward isn't compatible by lgray in https://github.com/CoffeaTeam/coffea/pull/794
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/799
* pins for coffea 2023.4.0rc3 by lgray in https://github.com/CoffeaTeam/coffea/pull/796
* Anticipate form.length_zero_array() highlevel deprecation by lgray in https://github.com/CoffeaTeam/coffea/pull/801
* Check for `Weight.partial_weight()` by rkansal47 in https://github.com/CoffeaTeam/coffea/pull/803
* build(deps): bump pypa/gh-action-pypi-publish from 1.8.5 to 1.8.6 by dependabot in https://github.com/CoffeaTeam/coffea/pull/804
* next coffea will be 2023.5.0.rc0 by lgray in https://github.com/CoffeaTeam/coffea/pull/802
* [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/805
* Repin for ak + dak + dhi by lgray in https://github.com/CoffeaTeam/coffea/pull/806
* The next coffea release will be 2023.5.0.rc1 by lgray in https://github.com/CoffeaTeam/coffea/pull/808
* fix: Allow lookup_base args to be in any order by lgray in https://github.com/CoffeaTeam/coffea/pull/811
* Use maintained action for conda and graphviz by lgray in https://github.com/CoffeaTeam/coffea/pull/812
* chore: remove ci based release by lgray in https://github.com/CoffeaTeam/coffea/pull/813
* chore: Update CONTRIBUTING.md to indicate calver, release process by lgray in https://github.com/CoffeaTeam/coffea/pull/815
* feat: pr labeller by lgray in https://github.com/CoffeaTeam/coffea/pull/816
* fix: use correct secret name by lgray in https://github.com/CoffeaTeam/coffea/pull/817
* fix(ci): no need to run outdated PR hashes by lgray in https://github.com/CoffeaTeam/coffea/pull/818
* build: dask-awkward/histogram 2023.5.1 by lgray in https://github.com/CoffeaTeam/coffea/pull/820
* feat: added rapidity getter in coffea/nanoevents/methods/vector.py by raeubaen in https://github.com/CoffeaTeam/coffea/pull/825
* build: move to pyproject style setup by lgray in https://github.com/CoffeaTeam/coffea/pull/819
* docs: Added example code for correctionlib_wrapper by yimuchen in https://github.com/CoffeaTeam/coffea/pull/790
* build: use hatch to build release instead of setup.py by lgray in https://github.com/CoffeaTeam/coffea/pull/828
* chore: update to empty_if_typetracer by lgray in https://github.com/CoffeaTeam/coffea/pull/821
* ci(pre-commit.ci): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/831
* fix: Added proper behavior for PFNanoAODSchema by yimuchen in https://github.com/CoffeaTeam/coffea/pull/832
* fix: serialize dask_awkward.Array to None and store hard ref in closure in behavior by lgray in https://github.com/CoffeaTeam/coffea/pull/824
* refactor: use length_zero/one_if_typetracer by lgray in https://github.com/CoffeaTeam/coffea/pull/834
* build: repin dask_awkward >= 2023.6.1 by lgray in https://github.com/CoffeaTeam/coffea/pull/836
* fix: PFnano format for dask by yimuchen in https://github.com/CoffeaTeam/coffea/pull/835
* feat: ML inference tools by yimuchen in https://github.com/CoffeaTeam/coffea/pull/798
* ci: [pre-commit.ci] pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/838
* fix: remove now-incorrect documentation from ml_tools by lgray in https://github.com/CoffeaTeam/coffea/pull/839
* feat: Pass list of files to NanoEventsFactory by chrispap95 in https://github.com/CoffeaTeam/coffea/pull/837
* build: pin to awkward 2.2.3, dask-histogram 2023.6.0 by lgray in https://github.com/CoffeaTeam/coffea/pull/840
* build(pre-commit.ci): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/841
* ci: get CI jobs completing again by lgray in https://github.com/CoffeaTeam/coffea/pull/842
* fix: adjustments to callable wrap to deal with typetracers in nested python structures by lgray in https://github.com/CoffeaTeam/coffea/pull/843
* ci: don't use url in test_fix823 by lgray in https://github.com/CoffeaTeam/coffea/pull/844
* chore: remove references to old awkward1 classes by lgray in https://github.com/CoffeaTeam/coffea/pull/846
* docs: update readme versions by lgray in https://github.com/CoffeaTeam/coffea/pull/847
* feat: New features to PackedSelection by iasonkrom in https://github.com/CoffeaTeam/coffea/pull/797
* ci: Bump pypa/gh-action-pypi-publish from 1.8.6 to 1.8.7 by dependabot in https://github.com/CoffeaTeam/coffea/pull/849
* build: pin awkward up to 2.2.4 by lgray in https://github.com/CoffeaTeam/coffea/pull/850
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/851
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/864
* Bump pypa/gh-action-pypi-publish from 1.8.7 to 1.8.8 by dependabot in https://github.com/CoffeaTeam/coffea/pull/866
* build: pin adjustments by lgray in https://github.com/CoffeaTeam/coffea/pull/852
* docs: Create CITATION.cff by nsmith- in https://github.com/CoffeaTeam/coffea/pull/853
* chore: coffea uses pyproject.toml now update in readme by lgray in https://github.com/CoffeaTeam/coffea/pull/868
* chore: Update README.rst with citation by nsmith- in https://github.com/CoffeaTeam/coffea/pull/869
* build: ignore dask-awkward 2023.8.0 by lgray in https://github.com/CoffeaTeam/coffea/pull/873
* ci: Bump pypa/gh-action-pypi-publish from 1.8.8 to 1.8.10 by dependabot in https://github.com/CoffeaTeam/coffea/pull/877
* fix: Make LumiMask compatible with dask-awkward by jrueb in https://github.com/CoffeaTeam/coffea/pull/879
* build: bump awkward pin to >= 2.3.3 by lgray in https://github.com/CoffeaTeam/coffea/pull/880
* feat: Dataset querying features using rucio by valsdav in https://github.com/CoffeaTeam/coffea/pull/883
* feat: dataset discovery CLI by valsdav in https://github.com/CoffeaTeam/coffea/pull/884
* ci: Bump actions/checkout from 3 to 4 by dependabot in https://github.com/CoffeaTeam/coffea/pull/889
* build: repin to latest fixed awkward by lgray in https://github.com/CoffeaTeam/coffea/pull/890
* feat: also pass `uproot_options` to `uproot.dask` in `from_root()` by iasonkrom in https://github.com/CoffeaTeam/coffea/pull/887
* fix: Daskify Elementlinks in PHYSLITE schema by nikoladze in https://github.com/CoffeaTeam/coffea/pull/872
* fix: allow for collections that contain non-jagged arrays in PHYSLITE schema by nikoladze in https://github.com/CoffeaTeam/coffea/pull/888
* ci: Bump crazy-max/ghaction-github-pages from 3 to 4 by dependabot in https://github.com/CoffeaTeam/coffea/pull/893
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/894
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/896
* fix: comply with `boost-histogram` 1.4.0 by iasonkrom in https://github.com/CoffeaTeam/coffea/pull/895
* ci: Bump amannn/action-semantic-pull-request from 5.2.0 to 5.3.0 by dependabot in https://github.com/CoffeaTeam/coffea/pull/897
* fix: Make rochester_lookup compatible with dask_awkward by jrueb in https://github.com/CoffeaTeam/coffea/pull/875
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/898
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/905
* feat: support `shape_touched` from Dask by agoose77 in https://github.com/CoffeaTeam/coffea/pull/900
* fix: added hasattr check to NanoAODEvents __repr__ by pviscone in https://github.com/CoffeaTeam/coffea/pull/909
* fix: remove binder setup problems in coffea 2023 by lgray in https://github.com/CoffeaTeam/coffea/pull/908
* ci: remove unnecessary python version warnings, reactivate nanoevents tests by lgray in https://github.com/CoffeaTeam/coffea/pull/912
* fix: test that demoted version of missing event id actually tries to extract from file by lgray in https://github.com/CoffeaTeam/coffea/pull/911
* build: unpin numpy from above, pin numba >= 0.58.1 by lgray in https://github.com/CoffeaTeam/coffea/pull/913
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/914
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/919
* ci: Bump amannn/action-semantic-pull-request from 5.3.0 to 5.4.0 by dependabot in https://github.com/CoffeaTeam/coffea/pull/920
* build: pin up to awkward 2.4.8, dask-awkward 2023.11.0 by lgray in https://github.com/CoffeaTeam/coffea/pull/921
* feat: switch permit_dask to True by default by lgray in https://github.com/CoffeaTeam/coffea/pull/922
* fix: remove all eager `len` calls by iasonkrom in https://github.com/CoffeaTeam/coffea/pull/928
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/927
* feat: add public `delta_r` and `delta_phi` kernels for user-specified phis and etas by iasonkrom in https://github.com/CoffeaTeam/coffea/pull/936
* ci: Bump conda-incubator/setup-miniconda from 2 to 3 by dependabot in https://github.com/CoffeaTeam/coffea/pull/938
* ci: Bump pypa/gh-action-pypi-publish from 1.8.10 to 1.8.11 by dependabot in https://github.com/CoffeaTeam/coffea/pull/941
* feat: Handle LumiData files that contain average instantaneous luminosity. by lgray in https://github.com/CoffeaTeam/coffea/pull/942
* ci: pin on major version for actions by lgray in https://github.com/CoffeaTeam/coffea/pull/943
* ci: pre-commit only has 3.0.0 by lgray in https://github.com/CoffeaTeam/coffea/pull/944
* feat: TreeMakerSchema Skimming (r2023) by yimuchen in https://github.com/CoffeaTeam/coffea/pull/935
* ci: Bump actions/setup-java from 3 to 4 by dependabot in https://github.com/CoffeaTeam/coffea/pull/946
* feat: use new dask methods by agoose77 in https://github.com/CoffeaTeam/coffea/pull/926
* fix: adjust dtypes for constants by lgray in https://github.com/CoffeaTeam/coffea/pull/948
* feat!: drop use of `opt_touch_all=False` by agoose77 in https://github.com/CoffeaTeam/coffea/pull/929
* chore: make dependabot semantic-titles friendly by lgray in https://github.com/CoffeaTeam/coffea/pull/947
* chore: fix gitter link by lgray in https://github.com/CoffeaTeam/coffea/pull/950
* fix: make `.metric_table()` and `.nearest()` work when `return_combinations` and `return_metric` are set to `True` respectively by iasonkrom in https://github.com/CoffeaTeam/coffea/pull/925
* fix: dispatch of _apply_global_index wasn't re-implemented correctly by lgray in https://github.com/CoffeaTeam/coffea/pull/951
* feat: adapt to dask from_map report feature by lgray in https://github.com/CoffeaTeam/coffea/pull/945
* test: taskvine+coffea+dask with pytest by btovar in https://github.com/CoffeaTeam/coffea/pull/952
* ci: bump actions/setup-python from 4 to 5 by dependabot in https://github.com/CoffeaTeam/coffea/pull/953
* fix: remove copyreg workaround, use attrs to store reference to original array by lgray in https://github.com/CoffeaTeam/coffea/pull/949
* fix(ci): only full version for publish by lgray in https://github.com/CoffeaTeam/coffea/pull/954
* build: depend on fsspec-xrootd for convenience by lgray in https://github.com/CoffeaTeam/coffea/pull/955
* fix: re-enable steps_per_file, warn about inefficiencies by lgray in https://github.com/CoffeaTeam/coffea/pull/956
* build(deps): pin to dask-awkward with reports by lgray in https://github.com/CoffeaTeam/coffea/pull/957
* fix: use awkward.mask instead of some_array.mask by lgray in https://github.com/CoffeaTeam/coffea/pull/959
* ci(pre-commit): pre-commit autoupdate by pre-commit-ci in https://github.com/CoffeaTeam/coffea/pull/958
* fix(ci): skip broken test in parsl by lgray in https://github.com/CoffeaTeam/coffea/pull/960
* fix: improvements to dataset_query tools by valsdav in https://github.com/CoffeaTeam/coffea/pull/940
* feat!: switch over to dask-based processing idioms, improve dataset handling by lgray in https://github.com/CoffeaTeam/coffea/pull/882
* build(deps): pin up to awkward 2.5.1, fsspec-xrootd 0.2.3 by lgray in https://github.com/CoffeaTeam/coffea/pull/962
* test: taskvine with remote env by btovar in https://github.com/CoffeaTeam/coffea/pull/964
* feat: Update delphes.py (DelphesSchema) by JaLuka98 in https://github.com/CoffeaTeam/coffea/pull/966
* fix: improved DataDiscoveryCLI interface by valsdav in https://github.com/CoffeaTeam/coffea/pull/965
* docs: prepare documentation for coffea 2023 by lgray in https://github.com/CoffeaTeam/coffea/pull/963
* chore: better defaults and typing by lgray in https://github.com/CoffeaTeam/coffea/pull/968
* build(deps): pin to uproot 5.2.0 by lgray in https://github.com/CoffeaTeam/coffea/pull/969
New Contributors
* rkansal47 made their first contribution in https://github.com/CoffeaTeam/coffea/pull/803
* raeubaen made their first contribution in https://github.com/CoffeaTeam/coffea/pull/825
* chrispap95 made their first contribution in https://github.com/CoffeaTeam/coffea/pull/837
* iasonkrom made their first contribution in https://github.com/CoffeaTeam/coffea/pull/797
* agoose77 made their first contribution in https://github.com/CoffeaTeam/coffea/pull/900
* pviscone made their first contribution in https://github.com/CoffeaTeam/coffea/pull/909
* JaLuka98 made their first contribution in https://github.com/CoffeaTeam/coffea/pull/966
**Full Changelog**: https://github.com/CoffeaTeam/coffea/compare/v0.7.21...v2023.12.0