Columnflow

- Hotfix sandbox interaction between tasks and array functions. (d7142d3a)
- Hotfix string representation. (52c4e824)
- 510: fix stitching (f92f77da)

- [195: Add task and functions to create ML training plots](https://github.com/columnflow/columnflow/issues/195)
- [273: Fix hiding of error bars.](https://github.com/columnflow/columnflow/pull/273)
- [263: Refactor call caches in TaskArrayFunctions.](https://github.com/columnflow/columnflow/pull/263)
- [277: Fix ratio plotting for MC stack with empty bins](https://github.com/columnflow/columnflow/pull/277)
- [276: Plotting data-mc ratio with empty MC bins leads to error](https://github.com/columnflow/columnflow/issues/276)
- [280: use true number of pileup interactions in pileup producer](https://github.com/columnflow/columnflow/pull/280)
- [282: Make category id columns configurable.](https://github.com/columnflow/columnflow/pull/282)
- [283: Raise awareness for priority of producers](https://github.com/columnflow/columnflow/issues/283)
- [285: Add Categorizer object.](https://github.com/columnflow/columnflow/pull/285)
- [286: Add checks for overlapping input columns.](https://github.com/columnflow/columnflow/pull/286)
- [287: Cache bundle repo requirement in HTCondorWorkflow.](https://github.com/columnflow/columnflow/pull/287)
- [288: Fix venv stacking by ignoring existing deps.](https://github.com/columnflow/columnflow/pull/288)
- [279: Modifying local repo files interferes with remote jobs](https://github.com/columnflow/columnflow/issues/279)
- [278: Empty `cf` sandbox](https://github.com/columnflow/columnflow/issues/278)
- [294: MemoryError: std::bad_alloc in CreateHistograms step ::SKELETON ANALYSIS](https://github.com/columnflow/columnflow/issues/294)
- [295: Software setup breaks on zsh](https://github.com/columnflow/columnflow/issues/295)
- [296: Support for crab jobs](https://github.com/columnflow/columnflow/issues/296)
- [297: External file access fails when inputs are staged-in inside sandboxes](https://github.com/columnflow/columnflow/issues/297)
- [298: Support for crab job submission.](https://github.com/columnflow/columnflow/pull/298)
- [300: disable call cache for Categorizers](https://github.com/columnflow/columnflow/pull/300)
- [302: Configure filling of empty bins in inference model.](https://github.com/columnflow/columnflow/pull/302)
- [281: Fix/nan pdf weights](https://github.com/columnflow/columnflow/pull/281)
- [303: Added MergeMLEvaluation task](https://github.com/columnflow/columnflow/pull/303)
- [305: Centralize types.](https://github.com/columnflow/columnflow/pull/305)
- [306: Add "--file-format {parquet,root}" option to UniteColumns](https://github.com/columnflow/columnflow/issues/306)
- [309: Optimize compression for parquet files](https://github.com/columnflow/columnflow/issues/309)
- [310: Allow writing root files in UniteColumns.](https://github.com/columnflow/columnflow/pull/310)
- [311: HTCondor jobs fail due to missing python package](https://github.com/columnflow/columnflow/issues/311)
- [312: Use ZSTD(1) as a default compression for all parquet files.](https://github.com/columnflow/columnflow/pull/312)
- [313: Enhance cmssw sandboxes](https://github.com/columnflow/columnflow/pull/313)
- [314: allow merging of MultiSettingsParameters and implement tests](https://github.com/columnflow/columnflow/pull/314)
- [315: Fix parsing of empty CSVParameter values](https://github.com/columnflow/columnflow/pull/315)
- [328: Improve memory usage of parquet files](https://github.com/columnflow/columnflow/issues/328)
- [329: Disable parquet dictionary encoding.](https://github.com/columnflow/columnflow/pull/329)
- [333: Update columnar software in venvs.](https://github.com/columnflow/columnflow/pull/333)
- [335: Fix root file export in UniteColumns.](https://github.com/columnflow/columnflow/pull/335)
- [338: Use dict encoding for mask columns](https://github.com/columnflow/columnflow/issues/338)
- [339: Use dict_encoding for selection results.](https://github.com/columnflow/columnflow/pull/339)
- [340: Change SelectionResult structure, add bool checks.](https://github.com/columnflow/columnflow/pull/340)
- [341: Added `PlotMLResults` Task to create CM](https://github.com/columnflow/columnflow/pull/341)
- [342: Ensure Python 3.7 compatibility.](https://github.com/columnflow/columnflow/pull/342)
- [343: Set proper LAW_CONFIG_FILE in remote jobs.](https://github.com/columnflow/columnflow/pull/343)
- [344: SelectionResult extension fails](https://github.com/columnflow/columnflow/issues/344)
- [345: Refactor chunked parquet reading](https://github.com/columnflow/columnflow/pull/345)
- [347: Add named event weight sets.](https://github.com/columnflow/columnflow/pull/347)
- [348: Introducing ColumnCollection's to define columns to keep.](https://github.com/columnflow/columnflow/pull/348)
- [349: Add empty selector](https://github.com/columnflow/columnflow/pull/349)
- [346: running Producer in cf.PrepareMLEvents](https://github.com/columnflow/columnflow/pull/346)
- [352: Add helper to select events from one category](https://github.com/columnflow/columnflow/issues/352)
- [369: Fix docs build, refactor remote env flags.](https://github.com/columnflow/columnflow/pull/369)
- [371: ROC curve implementation + tests for ML plotting](https://github.com/columnflow/columnflow/pull/371)
- [367: plot-function parameter not usable in PlotCutflowVariables1D](https://github.com/columnflow/columnflow/issues/367)
- [358: documentation build failed](https://github.com/columnflow/columnflow/issues/358)
- [384: Making MergeReducedEvents optional](https://github.com/columnflow/columnflow/issues/384)
- [388: Refactor dynamic requirements of reduced events.](https://github.com/columnflow/columnflow/pull/388)
- [404: allow masking events via preparation producer](https://github.com/columnflow/columnflow/pull/404)
- [405: request only one workflow per variable in CreateDatacards](https://github.com/columnflow/columnflow/pull/405)
- [402: Overflow Bins for Plotting](https://github.com/columnflow/columnflow/issues/402)
- [407: add plot_profile function](https://github.com/columnflow/columnflow/pull/407)
- [446: Trigger skip function creation in derived CSP subclasses.](https://github.com/columnflow/columnflow/pull/446)
- [448: Option to remove reduced events after merging.](https://github.com/columnflow/columnflow/pull/448)
- [449: Refactor pinning of output versions and locations.](https://github.com/columnflow/columnflow/pull/449)
- [450: Documentation](https://github.com/columnflow/columnflow/pull/450)
- [451: Add hooks to modify store parts on a per-task level.](https://github.com/columnflow/columnflow/pull/451)
- [453: Add markdown linting to CI](https://github.com/columnflow/columnflow/pull/453)
- [455: Handle empty LHE scale and pdf weight columns.](https://github.com/columnflow/columnflow/pull/455)
- [456: bug fix: was trying to convert jagged arrays to numpy views](https://github.com/columnflow/columnflow/pull/456)
- [458: Allow columns to be skipped in 'keep_columns' definitions.](https://github.com/columnflow/columnflow/pull/458)
- [460: correct bug in normalization setup when no cross section is available for a process](https://github.com/columnflow/columnflow/pull/460)
- [461: Improve process settings handling.](https://github.com/columnflow/columnflow/pull/461)
- [462: Process setting "scale=stack"](https://github.com/columnflow/columnflow/issues/462)
- [467: Accidentally overwriting dynamic requirements from producers](https://github.com/columnflow/columnflow/issues/467)
- [466: Fix histogram filling.](https://github.com/columnflow/columnflow/pull/466)
- [470: Add mechanism to hook into histograms (e.g. for ABCD methods)](https://github.com/columnflow/columnflow/pull/470)
- [471: Make bin shifting optional.](https://github.com/columnflow/columnflow/pull/471)
- [473: sort objects when passing wildcards](https://github.com/columnflow/columnflow/pull/473)
- [474: .copy not defined for awkward arrays. Use ak.copy instead](https://github.com/columnflow/columnflow/pull/474)
- [475: BUG: `tenacity` version 8.4.0 doesn't work](https://github.com/columnflow/columnflow/pull/475)
- [476: Linting the docs :)](https://github.com/columnflow/columnflow/pull/476)
- [468: allow stitching in normalization_weights Producer](https://github.com/columnflow/columnflow/pull/468)
- [469: Update cf and columnar sandboxes](https://github.com/columnflow/columnflow/issues/469)
- [479: Typo docs](https://github.com/columnflow/columnflow/pull/479)
- [480: Small fix for the `GetDatasetLFNs` task](https://github.com/columnflow/columnflow/pull/480)
- [482: Update sandboxes, adjust dak/ak IO.](https://github.com/columnflow/columnflow/pull/482)
- [481: allow building histograms in SelectEvents](https://github.com/columnflow/columnflow/pull/481)
- [485: Make el9 the default on NAF.](https://github.com/columnflow/columnflow/pull/485)
- [486: Add btag sf config, allowing for additional corrector args.](https://github.com/columnflow/columnflow/pull/486)
- [487: Feature: added a stack option for the scaling](https://github.com/columnflow/columnflow/pull/487)
- [488: Route brace expansion](https://github.com/columnflow/columnflow/pull/488)
- [489: FIX `stack` argument for the scaling of processes in the plotting](https://github.com/columnflow/columnflow/pull/489)
- [490: Plotting updates](https://github.com/columnflow/columnflow/pull/490)
- [492: Raise exception on missing cross sections in normalization weight producer](https://github.com/columnflow/columnflow/pull/492)
- [495: Scoped CSP requirements and inputs.](https://github.com/columnflow/columnflow/pull/495)
- [496: Fix ProvideReducedEvents](https://github.com/columnflow/columnflow/pull/496)
- [497: Collect CSP sandbox through dependencies.](https://github.com/columnflow/columnflow/pull/497)
- [499: add Selector for jet veto maps](https://github.com/columnflow/columnflow/pull/499)
- [501: Fix selector steps default resolution.](https://github.com/columnflow/columnflow/pull/501)
- [500: Fix and streamline jet veto maps selection.](https://github.com/columnflow/columnflow/pull/500)
- [505: Add TEC calibrator.](https://github.com/columnflow/columnflow/pull/505)
- [503: changed the way, the BRs of stitched processes are calculated](https://github.com/columnflow/columnflow/pull/503)
- [506: Fix input lfn localization in ReduceEvents](https://github.com/columnflow/columnflow/pull/506)
- [507: allow customizing repr of CSPWs](https://github.com/columnflow/columnflow/pull/507)

Enhancements and improvements

- Class aware decorators for task array functions (232)
- Allow histogramming jagged variables (233)
- Check that used/produced columns exist (240)
- Allow producer setup function to allow injecting files to be loaded in chunked event loop (242)
- add config and ml model to output path for MLTraining (237)
- add calib/sel/prod in MLTraining output parts (244)
- Producer groups (245)
- Reimplement coffea_parquet handler. (247)
- Limit MET pt in MET calibrator. (249)
- Add generic increment_stats selector. (253, 264)
- Add selection utils. (254)
- Add shift filters to csp decorators (255)
- Update columnar and config utils. (256)
- Allow custom cms label postfixes in plots. (257)
- Allow customizing default columnar sandbox (259)
- Skip calibration outputs and requirements dynamically. (260)
- Add skip_fn to create_category_combinations. (262)
- Read inputs for lambda-valued variable expressions. (265)
- Add functionality to mark used and produced columns as optional. (267)
- Add DeferredColumn. (269)
- Prefer calibrator(s), selector and producer(s) parameters from cli (268)
- Add process scale attribute to inference model (270)
- Improve version lookup in version map (272)


Breaking changes

The following updates are only related to changes in the software stack.

- Propagate local and global shift correctly to CSPs (258)
- Move from miniconda to micromamba. (246)
- Rename cf_prod venv to cf, improve dev_sandbox (261)


Fixes

- Fix software priority order (252)
- Uncomment uncertainty combination. (243)
- Handle negative and missing values in 2d plots (238)
- fix in MergeSelectionStats (231)
- Catch unkown process ids. (248)

Updates

143: enable direct input of numpy arrays
217: array function timing
212: Ensure unique sandbox install paths.
192: Add a config_util for automatic, deterministic category id generation
196: Add tasks to create yield tables
201: Different training and evaluation requirements for ML models
204: Add switch to auto-update sandboxes
210: Update default chunk size
211: Handle sandbox name collisions
219: Implementation of CreateYieldTable task
220: Add category id creation to config_util.
218: Speed up gen top decay producer.
206: Add fake weight for real data for cutflow handling.
205: Allow diverging training phase space
200: Make config entry access in producers/calibrators/selectors configurable.
208: Machine learning for multiple configs
227: Allow ML trainings to use data from multiple configs
224: change task outputs to dictionaries for calibration/selection/reduction


CMS specific

199: New selector for applying MET filters
198: New selector for applying data certification filter (golden JSON).
221: enabled handling of just eight LHEScaleWeights


Fixes

225: fix attachment of coffea behaviors
209: Fix empty setup mode for remote jobs.
216: Fix columns to read in ReduceEvents.

Updates

109: Update code style and venv requirements for Python 3.9.
107: Base software stack upon conda with python 3.9
38: Configure awkward_parquet to load specific nested columns
104: Declaring shift dependencies of event weights
91: Move residual analysis specific code
110: Add priority-based task queue to ChunkedReader.
112: Add default variables and categories to config.
113: Enable local processing
117: Update to awkward v2
106: Fast setup for newcomers
142: Add analysis template
134: Show category label in plots
141: Add hook to ml models for adding requirements.
144: Error with setup after awkward v2 upgrade
146: Tasks are run multiple times for the same effective_shift
147: Fix systematic shift resolution
148: Move cms related array functions and tasks to cms/ subdirectories.
145: Highlight cms-dependent tasks & producers
150: Take shape_norm into account for data/MC ratio.
149: Restructuring of plotting functions
151: Share prepared ML events between ML models.
154: Add mechanism to mark producers/calibrators as "mc_only"
156: Plotting throws DuplicateNameException when category has a channel
157: Use mc_only switches in producers / calibrators.
158: Fix task array function init in remote jobs.
159: Add first draft for config_utils.
152: Provide config_util.py
161: Handling arrays with zero rows/entries
180: Fix shift resolution.
162: Handling of empty chunks
182: Top-to-bottom ordering of stacked processes.
183: Remove need for multiple alias dicts per shift.
184: Modernize docs.
185: Use shallow od object copies in plotting.
186: Add config_util to create combinations of categories.
188: Support hiding error bars for unstacked processes.
187: Make plot tasks resolve categories after initializing dependent TaskArrayFunctions.
191: Fix order of array function dependency generation.
193: Make category_id producer aware of list of selectors
194: Handle lists of category selections in category_id producer.

https://github.com/orgs/uhh-cms/projects/3