Columnflow

Enhancements and improvements

- Class aware decorators for task array functions (232)
- Allow histogramming jagged variables (233)
- Check that used/produced columns exist (240)
- Allow producer setup function to allow injecting files to be loaded in chunked event loop (242)
- add config and ml model to output path for MLTraining (237)
- add calib/sel/prod in MLTraining output parts (244)
- Producer groups (245)
- Reimplement coffea_parquet handler. (247)
- Limit MET pt in MET calibrator. (249)
- Add generic increment_stats selector. (253, 264)
- Add selection utils. (254)
- Add shift filters to csp decorators (255)
- Update columnar and config utils. (256)
- Allow custom cms label postfixes in plots. (257)
- Allow customizing default columnar sandbox (259)
- Skip calibration outputs and requirements dynamically. (260)
- Add skip_fn to create_category_combinations. (262)
- Read inputs for lambda-valued variable expressions. (265)
- Add functionality to mark used and produced columns as optional. (267)
- Add DeferredColumn. (269)
- Prefer calibrator(s), selector and producer(s) parameters from cli (268)
- Add process scale attribute to inference model (270)
- Improve version lookup in version map (272)


Breaking changes

The following updates are only related to changes in the software stack.

- Propagate local and global shift correctly to CSPs (258)
- Move from miniconda to micromamba. (246)
- Rename cf_prod venv to cf, improve dev_sandbox (261)


Fixes

- Fix software priority order (252)
- Uncomment uncertainty combination. (243)
- Handle negative and missing values in 2d plots (238)
- fix in MergeSelectionStats (231)
- Catch unkown process ids. (248)

Updates

143: enable direct input of numpy arrays
217: array function timing
212: Ensure unique sandbox install paths.
192: Add a config_util for automatic, deterministic category id generation
196: Add tasks to create yield tables
201: Different training and evaluation requirements for ML models
204: Add switch to auto-update sandboxes
210: Update default chunk size
211: Handle sandbox name collisions
219: Implementation of CreateYieldTable task
220: Add category id creation to config_util.
218: Speed up gen top decay producer.
206: Add fake weight for real data for cutflow handling.
205: Allow diverging training phase space
200: Make config entry access in producers/calibrators/selectors configurable.
208: Machine learning for multiple configs
227: Allow ML trainings to use data from multiple configs
224: change task outputs to dictionaries for calibration/selection/reduction


CMS specific

199: New selector for applying MET filters
198: New selector for applying data certification filter (golden JSON).
221: enabled handling of just eight LHEScaleWeights


Fixes

225: fix attachment of coffea behaviors
209: Fix empty setup mode for remote jobs.
216: Fix columns to read in ReduceEvents.

Updates

109: Update code style and venv requirements for Python 3.9.
107: Base software stack upon conda with python 3.9
38: Configure awkward_parquet to load specific nested columns
104: Declaring shift dependencies of event weights
91: Move residual analysis specific code
110: Add priority-based task queue to ChunkedReader.
112: Add default variables and categories to config.
113: Enable local processing
117: Update to awkward v2
106: Fast setup for newcomers
142: Add analysis template
134: Show category label in plots
141: Add hook to ml models for adding requirements.
144: Error with setup after awkward v2 upgrade
146: Tasks are run multiple times for the same effective_shift
147: Fix systematic shift resolution
148: Move cms related array functions and tasks to cms/ subdirectories.
145: Highlight cms-dependent tasks & producers
150: Take shape_norm into account for data/MC ratio.
149: Restructuring of plotting functions
151: Share prepared ML events between ML models.
154: Add mechanism to mark producers/calibrators as "mc_only"
156: Plotting throws DuplicateNameException when category has a channel
157: Use mc_only switches in producers / calibrators.
158: Fix task array function init in remote jobs.
159: Add first draft for config_utils.
152: Provide config_util.py
161: Handling arrays with zero rows/entries
180: Fix shift resolution.
162: Handling of empty chunks
182: Top-to-bottom ordering of stacked processes.
183: Remove need for multiple alias dicts per shift.
184: Modernize docs.
185: Use shallow od object copies in plotting.
186: Add config_util to create combinations of categories.
188: Support hiding error bars for unstacked processes.
187: Make plot tasks resolve categories after initializing dependent TaskArrayFunctions.
191: Fix order of array function dependency generation.
193: Make category_id producer aware of list of selectors
194: Handle lists of category selections in category_id producer.

https://github.com/orgs/uhh-cms/projects/3

- Full zsh support (9f1185f, 1334f2e)
- Minor setup tweaks (057fae6, 3e03e89)
- Fix package setup (1c6bb50, 2f805a9, 693e523)
- Update example selector masks (845d779)

First release 🎉