Types-lxml

Features

- (82) Add buffer type support for upcoming lxml 6.0.
- `HtmlElement.text_content()` result will become plain `str` since lxml 6.0. This change shouldn't break much compatibility for users of previous lxml versions.
- Warn user about `str` input and `guess_charset` combo bug in `html.html5parser` functions
- Warn user about incorrect usage of specifying single element as `.extend()` argument
- lxml 6.0 exports `LIBXML_COMPILED_FEATURES` constant

Bug fixes

- (84) Tag selector supports iterator but not `bytearray`
- A few combinations of `QName` construction argument were actually disallowed; second argument can't be `QName` or `_Element` if first argument is non-empty
- Multiple issues for `Resolver` class
- Don't annotate opaque internal context object
- Drop `_ResolverRegistry.resolve()` which can't possibly appear in user land code
- Missing default value for `Resolver.resolve_file()` keyword arguments
- `Resolver.resolve()` arguments can be `None`
- Drop unused keyword arguments from `iterparse()` html mode overload
- `namespaces` arg of `.xpath()` method accepts tuple form. Change for `XPath` classes already done earlier.
- Confine the type of public element (subclass of `ElementBase`) class attributes
- `_Element.findtext()` didn't allow default argument in certain overload form
- `RelaxNG.from_rnc_string()` `base_url` argument accepts `bytes`
- `html.html5parser` `guess_charset` bug revisited
- `parse()` is not affected as it always open files/URL in binary mode
- For other functions, even `guess_charset=False` triggers the bug
- Some `html5parser.HTMLParser` initialisation arguments should be keyword only
- Corrected import of `typing.Never` in `html` module and `html.html5parser` submodule
- `.extend()` and `__setitem__()` of `_Element` and `HtmlElement` support iterator as value
- `_Element.index()` had wrong parameter name
- Continued verification of properties and arguments supporting `bytearray`:
- `_Element` `.text` and `.tail` properties
- Content-only elements
- `XPath` input expression
- `_IDDict` mixin arguments
- `xmlfile.write*()` methods and `encoding` argument

Minor changes

- Drop `_ElemClsLookupArg` alias, which is almost unused
- Rename `_StrictNSMap` to more aptly named `_StrOnlyNSMap`
- Don't include superclass attributes in `ParseError` definition
- Continue getting rid of `_AnyStr` in most places
- Mark constants as `Final`

Tests related

- Migrate following tests to property based runtime testing:
- All basic validators: `DTD`, `RelaxNG`, `ISO Schematron`
- All existing `_Element` method / property tests and content-only elements
- `html.html5parser` submodule
- `XMLID()` and friends
- `QName`
- For all negative tests on properties or arguments bombarded with random objects, also add iterables of correct objects to the list, to make sure iterables of correct argument or value would become incorrect arguments.

Documentation

- Fill in docstring for all `_Element` properties and methods

Features and breaking changes

- Depends on `beautifulsoup4` itself because version 4.13 has bundled inline annotation. Dropping `types-beautifulsoup4` dependency as result.
- Multi subclass patch includes change in `CSSSelector` result
- Implement `ErrorTypes` constants as enum

Bug fixes

- Additional `type: ignore`s that improve compatibility with older versions of `mypy` and `pyright`
- For `soupparser` submodule input arguments, copy definition from `beautifulsoup4` code directly
- `html.fragment_fromstring` `create_parent` argument can be string (83, thanks to sciyoshi)
- `XPath` `namespaces` argument can accept namespace tuples
- Fixes compatibility with mypy 1.14+
- `bytes` not allowed as `html.diff.htmldiff()` argument
- Parser `encoding` arguments do support `bytearray`
- `_ListErrorLog.filter_from_level()` supports real numbers

Minor changes and tests

- Migrate `beautifulsoup` and `ErrorLog` tests to property based
- Migrate `cssselect` and `XMLSchema` tests to runtime ones
- Add mocked HTTP response to file input fixture; introduces `urllib3` and `pook` as test dependency

Features and breaking changes

- Add [`basedpyright`](https://github.com/DetachHead/basedpyright) type checker support

- Incorporate changes from `lxml` 5.3.1 and (pending) 6.0
- More `html.builder` shorthands
- `libxml` feature constants
- `etree.DTD(external_id=...)` support `str` now
- Deprecate some `Memdebug` methods

Bug fixes

- `html.submit_form()` always return `HTTPResponse` for default handler

- Instance attributes are converted to properties because they are not deletable:
- `html.SelectElement.multiple`
- `html.InputElement.type`

- More function arguments supports `bytearray`:
- `register_namespace()`
- `inclusive_ns_prefixes` parameter of `etree.tostring()`

Minor changes

- Add docstring for some `etree` module function `overload`s
- Drop `_AnyStr` from `etree` module level functions

Breaking changes and features
- `bytearray` accepted as tag names, attribute names and attribute values
- Related change: create `_TextArg` type alias to slowly replace existing `_AnyStr` (71)
- Warn IDE users via `warnings.deprecated` about exception upon certain argument combinations in HTML link functions

Bug fixes
- Property deleter missing for HTML elements (73)
- `etree.strip_attributes()` support `bytes` and `QName` as input
- Completion of 64 for remaining known cases
- Corrected link replacement function return type in `html.rewrite_links()`
- `etree.canonicalize()` shouldn't accept `bytes` as input

Tests related
- Use [`hypothesis`](https://hypothesis.readthedocs.io/) for extensive tests on function arguments, currently used in `_Attrib` and HTML link function tests (#75)
- `reveal_type()` injector has been split into [its own project](https://github.com/abelcheung/pytest-revealtype-injector) and pulled via dependency

Internal changes
- Folder structure changes for the whole repository (70)
- Remove `_HANDLE_FAILURES` type alias and show values directly to users
- Rename type-only protocol `SupportsLaxedItems` to `SupportsLaxItems`

----
**Full Changelog**: https://github.com/abelcheung/types-lxml/compare/2024.11.08...2024.12.13

Breaking and important changes


- `pyright` users (and IDE that can make use of `pyright`) will see warning if a single string is supplied where collection of string is expected (`tuple`, `set`, `list` etc). In terms of typing, a single `str` itself is valid as a `Sequence`, so type checkers normally would not raise alarm when using `str` in such function parameters, but can induce unexpected runtime behavior. ([64](https://github.com/abelcheung/types-lxml/issues/64))
- `_ElementTree.write()`, `etree.fromstringlist()`, `etree.tostring()`, `html.soupparser.fromstring()`, `html.soupparser.parse()`
- It is possible to verify release files indeed come from GitHub and not maliciously altered. See [Release file attestation](README.mdrelease-file-attestation) for detail.
- Runtime tests support comparing with `mypy` results, therefore officially making static stub tests obsolete


Bug fixes

- Element tag names, attribute names and attribute values support `bytearray`. This is discovered via `hypothesis` testing, which is intended to be utilized in next release
- Compatibility with `pyright ⩾ 1.1.378`, which imposes additional overload warning for `etree.iterparse()`
- Use relative import in `lxml.ElementInclude`, otherwise `mypy` triggers `--install-type` behavior.
- `ObjectifiedElement` `__getitem()__` and `__setitem()__` should accept `str` as key, which behaves mostly like `__getattr__()` and `__setattr__()`. That means, `elem["foo"]` is equivalent to `elem.foo` for non-repeating subelements.

fixes for etree submodule

- `_Element.tag` property is not just a `str`. It is `str` after initial document or string parsing, but can be set manually to any type supported by tag name and returns the same object.
- When `QName` is initialized with first argument set to `None`, `_Element` can be used as second argument (which is promoted to first argument in implementation)
- Relax single argument usage in `_Element.iter*()` method family, doesn't need `tag=` keyword when argument is `None`
- `FunctionNamespace()` should generate an `_XPathFunctionNamespaceRegistry` object, not its superclass
- For [decorator usage](https://lxml.de/element_classes.html#implementing-namespaces-1) of `_XPathFunctionNamespaceRegistry` and `_ClassNamespaceRegistry`, decorator signature included an extraneous argument, though it doesn't affect any existing correct usage.
- `indent()` first parameter has wrong name

fixes for html submodule

- `soupparser.parse()` should accept `pathlib.Path` object as input
- `.value` property of `SelectElement` can't be set to `bytes`
- `.action` property of `FormElement` can have a value of `None`, and can be set to `None`. They have different meanings though.

Small and internal changes

- Declare python 3.13 support and perform CI tests.
- Separation of `pyright` and `mypy` ignore comments: in previous releases ` type: ignore[code]` was enabled in `pyright` settings. Now it only uses ` pyright: ignore[code]` so `mypy` comment won't affect `pyright` behavior.
- Add `._name` property to `html.FormElement` for form name
- Eliminate `typing.TypeAlias` usage ([declared obsolete](https://docs.python.org/3/library/typing.html#deprecation-timeline-of-major-features), and we can do without it)

Test related changes

- Stub tests migration to runtime:
- Most of remaining `etree._Element` methods, now only `.makeelement()` and `.xpath()` left in stub test
- Runtime test additions:
- `ElementNamespaceClassLookup()`
- `tox` config migrated to `pyproject.toml`, thus requiring `tox ⩾ 4.22`
- Runtime tests are now executed within `test-rt` folder due to python/mypy8400
- Some tests need to be performed conditionally when multi-subclass patch is applied
- Some tests or syntaxes need to be turned off to cope with `mypy` deficiencies
- Usage of [Rust-based `uv`](https://docs.astral.sh/uv/) as well as [related `tox` plugin](https://github.com/tox-dev/tox-uv) to speed up test environment recreation
- Don't force users installing [`tox-gh-actions`](https://github.com/ymyzk/tox-gh-actions) when checkout out repository, it is only useful for GitHub workflows

Docstring additions

- `etree` submodule: `parse()`, `fromstringlist()`, `tostring()`, `indent()`, `iselement()`, `adopt_external_document()`, `DocInfo` properties, `QName`, `CData`, some exception classes
- `html.soupparser` submodule: `fromstring()`, `parse()`, `convert_tree()`

Bug fix and small changes
================

- Namespace argument in Elementpath methods should allow `None` (60 thanks to cukiernick)

Internal changes
==========

- Perform runtime tests against `lxml 5.3`