Cdm-reader-mapper

------------------
Contributors to this version: Ludwig Lierhammer (:user:`ludwiglierhammer`) and Joseph Siddons (:user:`jtsiddons`)

New features and enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* implement both wrapper functions ``read`` and ``write`` that call the appropriate function based on ``mode`` argument (:pull:`238`):

* `mode` == "mdf"; calls ``cdm_reader_mapper.read_mdf``
* `mode` == "data"; calls ``cdm_reader_mapper.read_data`` or ``cdm_reader_mapper.write_data``
* `mode` == "tables"; calls ``cdm_reader_mapper.read_tables`` or ``cdm_reader_mapper.write_tables``

* optionally, call ``cdm_reader_mapper.read_tables`` with either source file or source directory path (:pull:`238`).

* apply attribute to ``DataBundle.data`` if attribute is nor defined in ``DataBundle`` (:pull:`248`).
* apply pandas functions directly to ``DataBundle.data`` by calling ``DataBundle.<pandas-func>`` (:pull:`248`).
* make ``DataBundle`` support item assignment for ``DataBundle.data`` (:pull:`248`).
* optionally, apply selections to ``DataBundle.mask`` in ``DataBundle.select_*`` functions (:pull:`248`).
* ``cdm_reader.reader.read_tables``: optionally, set null_label (:pull:`242`)
* new method function: ``DataBundle.select_where_all_false`` (:pull:`242`)
* new method functions: ``DataBundle.split_*`` which split a DataBundle into two new DataBundles containing data selected and rejected after user-defined selection criteria (:pull:`242`)

* ``DataBundle.split_by_boolean_true``
* ``DataBundle.split_by_boolean_false``
* ``DataBundle.split_by_column_entries``
* ``DataBundle.split_by_index``

* implement pandas indexer like ``iloc`` for not chunked data (:pull:`242`)

Internal changes
^^^^^^^^^^^^^^^^^

* ``cdm_reader_mapper.common.select``: restructure, simplify and summarize functions (:pull:`242`)
* split DataBundle class into main class (``cdm_reader_mapper.core._utilities``) and method function class (``cdm_reader_mapper.core.databundle``) (:pull:`242`)

Breaking changes
^^^^^^^^^^^^^^^^

* remove property ``tables`` from ``DataBundle`` object. Instead, ``DataBundle.map_model`` overwrites ``.DataBundle.data`` (:pull:`238`).
* set default ``overwrite`` values from ``True`` to ``False`` that is consistent with pandas ``inplace`` argument and rename ``overwrite`` to ``inplace`` (:pull:`238`, :pull:`248`).
* ``inplace`` returns ``None`` that is consistent with pandas (:pull:`242`)
* ``DataBundle`` method functions return a ``DataBundle`` instead of a ``pandas.DataFrame`` (:pull:`248`).
* ``DataBundle.select_*`` functions write only selected entries to ``DataBundle.data`` and do not take other list entries from ``common.select_*`` function returns into account (:pull:`248`).
* select functions do not reset indexes by default (:pull:`242`)
* rename ``DataBundle.select_*`` functions:

* ``DataBundle.select_true`` -> ``DataBundle.select_where_all_boolean``
* ``DataBundle.select_from_list`` -> ``DataBundle.select_where_entry_isin``
* ``DataBundle.select_from_index`` -> ``DataBundle.select_where_index_isin``

* rename ``cdm_reader_mapper.common.select_*`` functions and make them returning a tuple of selected and rejected data after user-defined selection criteria (:pull:`242`):

* ``select_true`` -> ``split_by_boolean_true``
* ``select_from_list`` -> ``split_by_column_entries``
* ``select_from_index`` -> ``spit_by_index``

Bug fixes
^^^^^^^^^

* ``cdm_reder_mapper.metmetpy``: set deck keys from ``???`` to ``d???`` in icoads json files which makes values accessible again (:pull:`238`).
* ``cdm_reder_mapper.metmetpy``: set ``imma1`` to ``icoads`` and ``immt`` to ``gcc`` in icoads/gcc json files which makes properties accessible again (:pull:`238`).
* ``DataBundle.copy`` function now makes a real deepcopy of ``DataBundle`` object (:pull:`248`).
* correct key index->section for self.df.attrs in open_netcdf (:pull:`252`)
* ``cdm_reader_mapper.map_model``: return null_label if conversion fails (:pull:`242`)
* keep indexes during duplicate check (:pull:`242`)

------------------
Contributors to this version: Ludwig Lierhammer (:user:`ludwiglierhammer`) and Joseph Siddons (:user:`jtsiddons`)

Announcements
^^^^^^^^^^^^^
This release drops support for Python 3.9 and adds support for Python 3.13 (:pull:`228`, :pull:`229`)

New features and enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* add environment.yml file (:pull:`229`)
* cdm_reader_mapper now separates the optional dependencies into dev and docs recipes (:pull:`232`).

* $ python -m pip install cdm_reader_mapper Install minimum dependency version
* $ python -m pip install cdm_reader_mapper[dev] Install optional development dependencies in addition
* $ python -m pip install cdm_reader_mapper[docs] Install optional dependencies for the documentation in addition
* $ python -m pip install cdm_reader_mapper[all] Install all the above for complete dependency version

Internal changes
^^^^^^^^^^^^^^^^
* GitHub workflow for ``testing_suite`` now uses ``uv`` for environment management, replacing ``micromamba`` (:pull:`228`)
* rename ci/requirements to CI and tidy up requirements/dependencies (:pull:`229`)

------------------
Contributors to this version: Ludwig Lierhammer (:user:`ludwiglierhammer`) and Joseph Siddons (:user:`jtsiddons`)

New features and enhancements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* New core ``DataBundle`` object including callable ``cdm_mapper``, ``metmemtpy`` and ``operations`` methods (:issue:`84`, :pull:`188`, :pull:`197`)
* Update readthedocs documentation (:issue:`191`, :pull:`197`)
* new function: ``write_data`` to write MDF data and validation mask according to ``write_tables`` for writing CDM tables (:pull:`201`)
* new function: ``read_data`` to read MDF data and validation mask according to ``read_tables`` for reading CDM tables (:pull:`201`)
* new property: DataBundle.encoding (:pull:`222`)
* add overwrite option to some DataBundel method functions (:pull:`224`)

Breaking changes
^^^^^^^^^^^^^^^^
* ``cdm_mapper``: ``map_model`` returns pandas.DataFrame instead of CDM dictionary (:pull:`189`)
* ``cdm_mapper``: rename function ``cdm_to_ascii`` to ``write_tables`` (:issue:`182`, :pull:`185`)
* ``cdm_mapper``: update parameter names and list of functions ``read_tables`` and ``write_tables`` (:pull:`185`)
* main ``cdm_mapper``, ``mdf_reader`` and ``duplicates`` modules are directly callable from ``cdm_reader_mapper`` (:pull:`188`)
* new list of imported submodules: [``map_model``, ``cdm_tables``, ``read_tables``, ``write_tables``, ``duplicate_check`` and ``read_mdf``] (:pull:`188`)
* removed list of imported submodules: [``cdm_mapper``, ``common``, ``mdf_reader``, ``metmetpy``, ``operations``] (:pull:`188`)
* remove imported submodules from ``cdm_mapper``, ``mdf_reader`` (:pull:`188`)
* ``read_tables``: returning ``DataBundle`` object (:pull:`188`)
* ``read_tables``: resulting dataframe always includes multi-indexed columns (:pull:`188`)
* ``duplicates`` is now a direct submodule of ``cdm_reader_mapper`` (:pull:`188`)
* import ``read`` function from ``mdf_reader.read`` as ``read_mdf`` (:pull:`188`)
* ``read_mdf``: returning ``DataBundle`` object (:pull:`188`)
* ``read_mdf``: remove parameter ``out_path`` to dump attribute information on disk (:pull:`201`)
* move function ``open_code_table`` from ``common.json_dict`` to ``cdm_mapper.codes.codes`` (:pull:``221`)
* ``operations`` to ``common`` (:pull:`224`)
* ``cdm_mapper``: rename table_writer to writer and table_reader to reader (:pull:`224`)
* ``mdf_reader``: rename write to writer and read to reader (:pull:`224`)
* ``metmetpy``: gather correction functions to correct module and validation functions to validate module (:pull:`224`)
* ``DataBundle``: remove properties selected, deselected, tables_dup_flagged and tables_dups_removed (:pull:`224`)

Internal changes
^^^^^^^^^^^^^^^^
* ``cdm_mapper``: dtype conversion from ``write_tables`` to new submodule ``_conversions`` of ``map_model`` (:pull:`189`)
* ``cdm_mapper``: rename ``mappings`` to ``_mapping_functions`` (:pull:`189`)
* ``cdm_mapper``: mapping functions from ``mapper`` to new submodule ``_mappings`` (:pull:`189`)
* ``cdm_mapper``: save utility functions from ``table_reader.py`` and ``table_writer.py`` to ``_utilities.py`` (:pull:`185`)
* reduce complexity of several functions (:issue:`25`, :pull:`200`):

* ``mdf_reader.read.read``
* ``mdf_reader.validate.validate``
* ``mfd_reader.utils.decoders.signed_overpunch``
* ``cdm_mapper._mappings._mapping``
* ``metmetmpy.station_id.validate.validate``

* split ``mdf_reader.utils.auxiliary`` into ``mdf_reader.utils.filereader``, ``mdf_reader.utils.configurator`` and ``mdf_reader.utils.utilities`` (:issue:`25`, :pull:`200`)
* simplify ``cdm_mapper.read_tables`` function (:pull:`192`)
* ``mdf_reader``: Refactored ``Configurator`` class, ``Configurator.open_pandas`` method, to handle looping through rows (:pull:`208`, :pull:`210`)
* ``mdf_reader``: Refactored ``Configurator`` class, ``Configurator.open_data`` method, to avoid creating a pre-validation missing_value mask (:pull:`216`)
* ``mdf_reader``: move ``validate`` to ``utils.validators`` (:pull:`216`)
* ``mdf_reader``: no need for multi-column key codes (e.g. ``("core", "VS")``) (:pull:`221`)
* ``mdf_reader.utils.validator``: simplify function ``code_validation`` (:pull:`221`)
* ``cdm_mapper.codes.common``: convert range-key properties to list (:pull:`221`)
* ``testing_suite``: new chunksize test with icoads_r300_d721 (:pull:`222`)
* ``mdf_reader``, ``cdm_nmapper``: use model-depending encoding while writing data on disk (:pull:`222`)
* code restructuring (:pull:``224`)
* remove unused functions and methods (:pull:`224`)


Bug fixes
^^^^^^^^^
* Solve SettingWithCopyWarning (:issue:`151`, :pull:`184`)
* ``mdf_reader``: ``utils.converters.decode`` returns values not only None (:pull:`214`)
* ``mdf_reader``: solving misleading reading due to German "umlauts"(:issue:`212`, :pull:`214`, :pull:`222`)

------------------
Contributors to this version: Ludwig Lierhammer (:user:`ludwiglierhammer`)

Announcements
^^^^^^^^^^^^^
* New PyPi Classifiers:

* Development Status :: 5 - Production/Stable
* Development Status :: Intended Audience :: Science/Research
* License :: OSI Approved :: Apache Software License
* Operating System :: OS Independent

------------------
Contributors to this version: Ludwig Lierhammer (:user:`ludwiglierhammer`)

Announcements
^^^^^^^^^^^^^
* set package version to v1.0.1

------------------
Contributors to this version: Ludwig Lierhammer (:user:`ludwiglierhammer`)

Announcements
^^^^^^^^^^^^^
* Final version used for GLAMOD marine processing release 7.0

Bug fixes
^^^^^^^^^
* ``cdm_mapper``: Two reports that describe each other as best duplicates are not flagged as duplicates (DupDetect) (:pull:`149`)
* ``cdm_mapper``: Reindex only if null values available (DupDetect) (:pull:`153`)