Sphinx-needs

:Full Changelog: `v3.0.0...v4.0.0 <https://github.com/useblocks/sphinx-needs/compare/3.0.0...4.0.0>`__

Breaking Changes
................

This commit contains a number of breaking changes:

Improvements to filtering at scale
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For large projects, the filtering of needs in analytical directives such as :ref:`needtable`, :ref:`needuml`, etc, can be slow due to
requiring an ``O(N)`` scan of all needs to determine which to include.

To address this, the storage of needs has been refactored to allow for pre-indexing of common need keys, such as ``id``, ``status``, ``tags``, etc, after the read/collection phase.
Filter strings such as ``id == "my_id"`` are then pre-processed to take advantage of these indexes and allow for ``O(1)`` filtering of needs, see the :ref:`filter_string_performance` section for more information.

This change has required changes to the internal API and stricter control on the access to and modification of need data, which may affect custom extensions that modified needs data directly:

- Access to internal data from the Sphinx `env` object has been made private
- Needs data during the write phase is exposed with either the read-only :class:`.NeedsView` or :class:`.NeedsAndPartsListView`, depending on the context.
- Access to needs data, during the write phase, can now be achieved via :func:`.get_needs_view`
- Access to mutable needs should generally be avoided outside of the formal means, but for back-compatibility the following :external+sphinx:ref:`Sphinx event callbacks <events>` are now available:

- ``needs-before-post-processing``: callbacks ``func(app, needs)`` are called just before the needs are post-processed (e.g. processing dynamic functions and back links)
- ``needs-before-sealing``: callbacks ``func(app, needs)`` just after post-processing, and before the needs are changed to read-only

Additionally, to identify any long running filters,
the :ref:`needs_uml_process_max_time`, :ref:`needs_filter_max_time` and :ref:`needs_debug_filters` configuration options have been added.

Key changes were made in:

- ♻️ Replace need dicts/lists with views (with fast filtering) in :pr:`1281`
- 🔧 split ``filter_needs`` func by needs type in :pr:`1276`
- 🔧 Make direct access to ``env`` attributes private in :pr:`1310`
- 👌 Move sorting to end of ``process_filters`` in :pr:`1257`
- 🔧 Improve ``process_filters`` function in :pr:`1256`
- 🔧 Improve internal API for needs access in :pr:`1255`
- 👌 Add ``needs_uml_process_max_time`` configuration in :pr:`1314`
- ♻️ Add ``needs_filter_max_time`` / ``needs_debug_filters``, deprecate ``export_id`` in :pr:`1309`

Improved warnings
~~~~~~~~~~~~~~~~~

sphinx-needs is designed to be durable and only except when absolutely necessary.
Any non-fatal issues during the build are logged as Sphinx warnings.
The warnings types have been improved and stabilised to provide more information and context, see :ref:`config-warnings` for more information.

Additionally, the :func:`.add_need` function will now only raise the singular exception :class:`.InvalidNeedException` for all need creation issues.

Key changes were made in:

- 👌 Warn on unknown need keys in external/import sources in :pr:`1316`
- ♻️ Extract ``generate_need`` from ``add_need`` & consolidate warnings in :pr:`1318`

Improved ``needs.json``
~~~~~~~~~~~~~~~~~~~~~~~

A number of output need fields have been changed, to simplify the output.
Key changes were made in:

- 🔧 change type of need fields with ``bool | None`` to just ``bool`` in :pr:`1293`
- ♻️ Remove ``target_id`` core need field in :pr:`1315`
- ♻️ Output ``content`` in ``needs.json`` not ``description`` in :pr:`1312`
- 👌 Add ``creator`` key to ``needs.json`` in :pr:`1311`

Replacement of ``[[...]]`` and ``need_func`` in need content
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The parsing of the ``[[...]]`` dynamic function syntax in need content could cause confusion and unexpected behaviour.
This has been deprecated in favour of the new, more explicit :ref:`ndf role <ndf>`, which also deprecates the ``need_func`` role.

See :pr:`1269` and :pr:`1266` for more information.

Removed deprecation
~~~~~~~~~~~~~~~~~~~

The deprecated ``needfilter`` directive is now removed (:pr:`1308`)

New and improved features
.........................

- ✨ add ``tags`` option for ``list2need`` directive in :pr:`1296`
- ✨ Add ``ids`` option for ``needimport`` in :pr:`1292`
- 👌 Allow ``ref`` in ``needuml`` to handle need parts in :pr:`1222`
- 👌 Improve parsing of need option lists with dynamic functions in :pr:`1272`
- 👌 Improve warning for needextract incompatibility with external needs in :pr:`1325`
- 🔧 Set ``env_version`` for sphinx extension in :pr:`1313`

Bug Fixes
.........

- 🐛 Fix removal of ``Needextend`` nodes in :pr:`1298`
- 🐛 Fix ``usage`` numbers in ``needreport`` in :pr:`1285`
- 🐛 Fix ``parent_need`` propagation from external/imported needs in :pr:`1286`
- 🐛 Fix ``need_part`` with multi-line content in :pr:`1284`
- 🐛 Fix dynamic functions in ``needextract`` need in :pr:`1273`
- 🐛 Disallow dynamic functions ``[[..]]`` in literal content in :pr:`1263`
- 🐛 fix parts defined in nested needs in :pr:`1265`
- 🐛 Handle malformed ``filter-func`` option value in :pr:`1254`
- 🐛 Pass ``needs`` to ``highlight`` filter of ``graphviz`` `needflow` in :pr:`1274`
- 🐛 Fix parts title for ``needflow`` with ``graphviz`` engine in :pr:`1280`
- 🐛 Fix ``need_count`` division by 0 in :pr:`1324`

* Improvement: Fixed :ref:`needextend` error handling by adding a strict-mode option to it.
(:issue:`747`)
* Improvement: Fixed issue with handling needs-variants by default.
(:issue:`776`)
* Improvement: Performance fix needs processing.
(:issue:`756`)
* Improvement: Performance fix for needflow.
(:issue:`760`)
* Improvement: Fixed rendering issue with the debug layout.
(:issue:`721`)
* Improvement: Added :ref:`needs_show_link_id`.
* Improvement: Supported arguments as filter string for :ref:`needextract`.
(:issue:`688`)
* Improvement: Added :ref:`needs_render_context` configuration option which enables you to use custom data as the
context when rendering Jinja templates or strings.
(:issue:`704`)
* Improvement: Supported ``target_url`` for :ref:`needs_external_needs`.
(:issue:`701`)
* Bugfix: Fixed needuml key shown in need meta data by providing internal need option `arch`.
(:issue:`687`)
* Improvement: Included child needs inside their parent need for :ref:`needflow`.
(:issue:`714`)
* Improvement: Supported generate need ID from title with :ref:`needs_id_from_title`.
(:issue:`692`)
* Improvement: Supported download ``needs.json`` for :ref:`needimport`.
(:issue:`715`)
* Bugfix: Fixed import() be included in needarch.
(:issue:`730`)
* Bugfix: Needuml: uml() call circle leads to an exception :ref:`needarch_ex_loop`.
(:issue:`731`)
* Improvement: needarch provide need() function to get "need data".
(:issue:`732`)
* Improvement: needuml - flow() shall return plantuml text without newline.
(:issue:`737`)
* Bugfix: Needuml used but "sphinxcontrib.plantuml" not installed leads to exception
(:issue:`742`)
* Improvement: better documentation of mixing orientation and coloring in needs_extra_links
(:issue:`764`)
* Bugfix: Needarch: Fixed import() function to work with new implemented flow() (737).
(:issue:`752`)
* Bugfix: Needtable: generate id for nodes.table
(:issue:`434`)
* Improvement: Updated pantuml in test folder to same version as in doc folder
(:issue:`765`)

* Improvement: Schema check for ``need.json`` files implemented.
* Improvement: New option for ``needtable`` and co: :ref:`filter_func`, which allows to reference and use python code
as filter code from external files
(:issue:`340`)
* Bugfix: Fixed :ref:`needs_builder` handling warnings about missing needs.json when :ref:`needs_file` not configured
(:issue:`340`)
* Bugfix: unstable build with :ref:`needs_external_needs`
(:issue:`399`)
* Bugfix: :ref:`needs_external_needs` reads external need status now and warnings gets not checked for
:ref:`needs_external_needs`
(:issue:`375`)

* Improvement: New config option :ref:`needs_builder_filter` to define a filter for the needs builder.
(:issue:`342`)
* Improvement: Added option ``json_path`` for :ref:`needs_external_needs` to support external needs from local ``needs.json`` files.
(:issue:`339`)
* Improvement: Providing :ref:`needs_table_classes` to allow to set custom table css classes, to better support
themes like ReadTheDocs.
(:issue:`305`)
* Improvement: Supporting user defined filter code function for :ref:`needs_warnings`
(:issue:`345`)
* Improvement: Supporting caption for :ref:`needtable`
(:issue:`348`)
* Improvement: New config option :ref:`needs_filter_data` to allow to use custom data inside a :ref:`filter_string`
(:issue:`317`)
* Improvement: API to register warnings
(:issue:`343`)
* Bugfix: Scrolling tables improved and ReadTheDocs Tables fixed
(:issue:`305`)
* Bugfix: :ref:`needtable` need parts 'id' column is not linked
(:issue:`336`)
* Bugfix: :ref:`needtable` need parts 'incoming' column is empty
(:issue:`336`)
* Bugfix: :ref:`needs_warnings` not written to error log.
(:issue:`344`)
* Improvement: Providing :ref:`needs_warnings_always_warn` to raise sphinx-warnings for each not passed :ref:`needs_warnings`.
(:issue:`344`)
* Bugfix: :ref:`needimport` relative path not consistent to Sphinx default directives.
(:issue:`351`)

:Full Changelog: `v2.0.0...v2.1.0 <https://github.com/useblocks/sphinx-needs/compare/2.0.0...2.1.0>`__

Improvements
............

- 👌 Default to warning for missing ``needextend`` ID in :pr:`1066`
- 👌 Make ``needtable`` titles more permissive in :pr:`1102`
- 👌 Add ``filter_warning`` directive option, to replace default warning text in :pr:`1093`
- 👌 Improve and test github ``needservice`` directive in :pr:`1113`
- 👌 Improve warnings for invalid filters (add source location and subtype) in :pr:`1128`
- 👌 Exclude external needs from ``needs_id_regex`` check in :pr:`1108`
- 👌 Fail and emit warning on filters that do not return a boolean result in :pr:`964`
- 👌 Improve ``Need`` node creation and content parsing in :pr:`1168`
- 👌 Add loading message to ``permalink.html`` in :pr:`1081`
- 👌 Remove hard-coding of ``completion`` and ``duration`` need fields in :pr:`1127`

Bug fixes
.........

- 🐛 Image layout function in :pr:`1135`
- 🐛 Centralise splitting of need ID in :pr:`1101`
- 🐛 Centralise need missing link reporting in :pr:`1104`

Internal improvements
......................

- 🔧 Use future annotations in all modules in :pr:`1111`
- 🔧 Replace black/isort/pyupgrade/flake8 with ruff in :pr:`1080`

- 🔧 Add better typing for ``extra_links`` config variable in :pr:`1131`
- 🔧 Centralise need parts creation and strongly type needs in :pr:`1129`
- 🔧 Fix typing of need docname/lineno in :pr:`1134`
- 🔧 Type ``ExternalSource`` config dict in :pr:`1115`
- 🔧 Enforce type checking in ``needuml.py`` in :pr:`1116`
- 🔧 Enforce type checking in ``api/need.py`` in :pr:`1117`
- 🔧 Add better typing for ``global_options`` config variable in :pr:`1120`

- 🔧 Move dead link need fields to internals in :pr:`1119`
- 🔧 Remove usage of ``hide_status`` and ``hide_tags`` in :pr:`1130`
- 🔧 Remove ``hidden`` field from ``extra_options`` in :pr:`1124`
- 🔧 Remove ``constraints`` from ``extra_options`` in :pr:`1123`
- 🔧 Remove use of deprecated ``needs_extra_options`` as ``dict`` in :pr:`1126`

* Bugfix: Fixed pygls version compatibility.
(:pr:`867`,
:pr:`865`)