Micropython-stubber

Naming convention
- use {family}-v{version}-{port} notation across all scripts and tools
- updated documentation accordingly
- single function to handle version names in various formats (clean_version)
- requirements: pin dependencies to avoid differences when running acriss multiple machines.
- get_mpy_frozen:
- refactor module
- checkout the matching commit of micropython-lib
withouth this it is not possible to build stubs older versions due to a restructure of micropython-lib
- bulk_stubber:
- make more robust by including depedencies in the project.
- detect pyboard based on USB VID & PID
- prep autostubber for pybv11
- manifest.json generation
- improve core manifest.json information
- improve support for grouping and sorting


stubber cli:
- add init test & refector imports
- fix tests
- merge docstubs into stubber
- reduce tests
- merge get-all-stubs into stubber
- merge get_all_stubs into stubber
- add init
- add stub and some tests
- refactor minify and add -all option

config
- change to use module tomlli(b)
common:
- update pyright config to reduce noise

firmware stubber
fw-stubber: clean up excepts

Documentation
docs: fix tomllib
docs: document the CLI for stubber using sphinx-click
readme: add badges for pypi and codecov
doc updates
fix docbuild for poetry
update totdos in source

docstubs: fix json import
fix: black formatting across platforms

add settings and script for coverage reporting
update snippets
(origin/dev/snippets) snippets:add checks

Develop:
fix: codespace poetry setup

test
test: fix fewer arguments
tests: add firmware stubber version tests
test: add missing toml
test: add package == version == firmware stubber version tests
test: basicgit - add mocktest git
test: fix minify test remove unused imports

Github Actions:
wf: use poetry run and poetry install

pkg: updates to support poetry
wf: poetry install verbose
pkg: update actions to work with poetry
pkg: Move all stubber files into module folder
pkg: get version from poetry package
pkg: move csv into package
config: remove pylance config from vscode settings
pkg: add data files

Use Poetry

Dependencies
- bump distro from 1.6.0 to 1.7.0
- bump pytest from 6.2.5 to 7.0.1
- bump myst-parser from 0.16.1 to 0.17.0
- bump coverage from 6.3 to 6.3.1
- bump black from 21.12b0 to 22.1.0
- bump libcst from 0.3.23 to 0.4.1
- bump pytest-mock from 3.6.1 to 3.7.0
- bump mpy-cross from 1.17 to 1.18

documentation
- Add notes to docstrings/descriptions
- (bluetooth_constants) doc: configure submodules
- docs: add open in VSCode banner
- textual and updates to version notation
- Update 40_firmware_stubs.md

docstubs:
- fix optional - Optional Parameters are not treated as optional https://github.com/Josverl/micropython-stubber/issues/183
- add docstubs validation
- skip 3 stubs.
- create umodules https://github.com/Josverl/micropython-stubber/issues/176
- allign to micropython PR and update tests
- workaround for re.Match and smarter tests
- fix module & Class constant handling & tests
- Add `= ...` to module and class level constants to avoid errors when they are used as a default value in function of nethod

scripts:
- script update_modulelist, - write updates to: - board/modulelist.txt - board/createstubs.py
- bulk_stubber: Copy generated/firmwarestubs to all-stubs/* use firmware-stubs path for temp storage
- Update BulkStubber & getserial for esp32, esp8266 and stm32
- minify.ps1: posix paths
- bulkstubber: more versions

Common
- improve black formatting for subfolders
- utils: do not use BaseException
- config: pycache
- data: add firmware modules
- Multi-root with mpy-stdlib based on pico-go
- utils.stubgen: re-apply refactor to avoid use of os.cmd or subprocess.run
- utils.stubgen: refactor to avoid use of os.cmd or subprocess.run

Validation
- add snippets to test mpy stdlib
- validation: Add more snippets
- update validation snippets
- WS with pyright snippets

Github Actions:
- wf: do not run minified tests twice, minify all

Developing:
devcontainer: also init git submodules
devcontainer: change python setup

Tests
test: fix test_socket _class
test: fix return type and test deepsleep
test: skip test_createstubs on windows + python 3.7
test: native test on windows, include db test on linux
test: add createstubs_db on all platforms
test: lower theshold to 25 stubs
test: add native integration test for micropython_mem
test: linux rest on ubuntu and debian, seperate branch / fail logic in docstubs test
test: add additional firmwares
test: waste less time in by reducing test size
test: remove path test
debug: fix debug config for tests
debug: add missing debug property
test: fix pessimistic test and make more robust

Better exeptions fix missed updates to mem and stub variants

Change naming convention:
- board: user {family}-v{version}-{port} notation
- change name of master branch to main

stubber cli
- fix: - rp2.PIO.irq
- fix dequeue
- revert re.match change
- fixup re.match
- add fixes for remaining Documentation errors
- generate Exception Classes
- fix test and default for machine.Pin.__init__
- fix 3 Non-default argument follows default argument errors
- run pyflake to remove unused imports
- fix black parameters
- no need to redefine Exception
- use 'Latest' as a version tag for the most recent master
- use v for version


Common
- pyright: restore exec environments test & board
- get_frozen: improve force __init__.py
- get_frozen: run black on new/updated stubs
- createstubs: support for RP2 , and accept kwargs for methods and functions
- lobo: skip some modules
- get_mpy: match_lib - add v1.18 rename to .csv
- version: latest
- basicgit: remove debugging code
- basicgit: fix gettag 'latest'
- bulk_stubber: make more robust by adding the dependencies to the repo
- get_frozen: clean collected modules - freeze to empty directory - block CI/CD manifest
- get_mpy: for frozen modules: checkpout the matching commit of micropython-lib
- mpy_frozen: refactor module
- process: --source allows scriptname to be specified & add mpy-cross step
- update_pyi : rename and more efficient updating find .pyi`s created in the wrong location and move them
- Manifiest: release is optional
- use __version__ and bump version
- get_mpy: improve porcessing of frozen modules V1.16 and newer
- schema: change stubtype property
- Module manifest: Sorting and add port name
- update get-frozen
- improve frozen manifest processing (88)
- utils: update clean version
- createstubs* : Add stubtype to manifest - remove redundant try/catch
- process.py: cleanup unused variables
- process.py: use click

scripts
- add scripts to simplify copy and updates
- Merge generate stubs from documentation

Firmware stubber
- createstubs: sample bootfile works on pyb & esp
- createstubs: get_root can detect /sd cards

Dependencies
- python requirements: pin versions
- bump coverage from 6.2 to 6.3
- bump sphinx from 4.3.2 to 4.4.0
- add autoflake
- bump mpy-cross from 1.16 to 1.17
- bump myst-parser from 0.15.2 to 0.16.1
- bump mypy from 0.930 to 0.931
- bump sphinx from 4.3.1 to 4.3.2
- bump rshell from 0.0.30 to 0.0.31

Documentation
- docs: update naming convention
- fix stuborder image
- Update 10_approach.md
- Naming convention
- one less
- add branch rename instructions
- documented Docs to Stubs process

validation
- basic setup for stub validation
- add code snippets for validation

BugFixes:
- fix: ensure that `from __future__ ...` is at start
- add .pyi stubs for umqtt.robust Workaround for missing __init__.py in source
- fix: allow stubgen of async yield in stub (python 3.6) closes 137
- fix Exception lineskip
- fix: get_mpy - save&resore cwd
- fix lobotest to match reduced modules
- Fix core module sort order closes 68


Github Actions:
- wf test: ensure artefact upload on error
- wf: report coverage
- testspace: use folders
- pytest: upload to testspace
- wf: allign naming

Scripts:
- prep autostubber for pybv11
- detect pyboard

code quality:
- fix warnings
- improve core manifest.json

Tests
- test: fix testregression for latest version
- tests: docstubs improve test of class documented as function
- testspace script
- test: add mpy-cross test and split to seperate folder
- improve grouping
- pytest - use matrix.os in test results
- test: do not run pytest-cov
- tests: drop utf-8 encoding
- test : init logging
- test : fix path
- test: stabelize pyright & black detection
- tests: skip basicgit tests
- add integration tests for cmdline



improve docstubs
- rst: cleaner import though use of __all__
- document: debug subProcess
- docstubs: fix random.choice(): Any
- docstubs: workaround black on Py3.7

common:
- add header to modulelist
- devcontainer: simpler requirements
- utils: fix generating .pyi files from .py files with errors
- fix: remove duplication fix minor issues
- github: group logging

Tests
- pin python version
- pytest: add test markers
- change ubuntu version detection to codename
- add mark.minfied to linux tests add new platform markers
- implement workaround for failing minified tests
- tests: restructure board test to share testdata
- add debug config
- tests: fix minified tests
- tests: improve & merge testing
- test: linux detection++
- tests: use pathlib and fix paths
- clarify intentional error
- remove duplicate tests
- tests: Restucture Micropython on Cpython tests
- tests: resolve issues due to sloppy imports in tests
- stubgen test : add logging for python 3.7

minify:
- minify all variants
- also remove _log * lines
- also: minimize createstubs_mem.py
- Fix: remove duplicate builtins from minified
- minify : fix mpy-cross on Python 3.7

createstubs:
- add normal and mem_constrained versions
- add database variant
- Save state to database and reboot on memory error
- reduce complexity from createstubs to save ram possibly loosing some edge cases ++ docs
- update to handle multi-file uploads to overcome esp8266 memory constraints
- both normal and db work on esp8266 (report not done)
- keep get_root and _info to allow for testing
- update tests to clean-up sys.path after the tests
- keep __version__ in minified
- use __version__ move unneeded import of machine
- gracefully handle missing `modules.txt`
- stubber: fix firmwareid for mpy esp8266
- stub_lvgl: fix __version__


Github Actions:
- use posix path
- workflow: test : on checkout fetch all branches and tags

Docs:
- do not keep generated API Docs
- refactor documenation to more docs
- add explanation om memory optimization.
- add cloning of submodules
- safer path insertion

Other :
- add remote_stubber script
- bulk_stubber: run black formatter after all downloads
- remote_stubber: improve reporting
- bulk stubber: make esp8266 work more reliable ESP32 WIP - scrips fail to upload

- get_root can detect /sd cards
- sample bootfile works on pyb1.1, esp323, esp8266

Documentation Stubs

- avoid the use of BaseException

- significant memory optimisation for use on low-memory devices such as the esp8266 family
- load the list of modules to be stubbed from a text file rather than as part of the source
- use both minification and the `mpy-cross` compiler to reduce the claim on memory (RAM)

{warning}
This is a potential breaking change for external tools that expect to either directly execute the script or upload only a single file to an MCU in order to stub.

- the current process is automated in [`remote_stubber.ps1'][remote_stubber]

- Fixes a regression introduced in 1.4-beta where function definitions would include a self parameter.

- Switched to use [python-minifier](https://github.com/dflook/python-minifier) for the minification due to the end-of-life of the previous minification tool
The new minification tool produces more compact code, although that is still not sufficient for some memory constrained devices.
- there are no functional changes,
- the detection of Micropython was adjusted to avoid the use of eval which blocked a minification rule
- several tests were adjusted

documentation
- Add Sphinxs documentation
- changelog
- automatic API documentation for
* createstubs.py (board)
* scripts to run on PC / Github actions
- Publish documentation to readthedocs