Mcpyrate

**Changed**:

- Small improvements to unparser:
- No space after unary `+`, `-` or `~`.
- Future-proofing: yell if an unsupported constant value type is encountered.

- Add a new troubleshooting item on another [Heisenbug](https://en.wikipedia.org/wiki/Heisenbug) that can occur when buggy macros are used inside a `with step_expansion`.


---

**Changed**:

- Documentation improved. Particularly, AST markers are now documented (in the main user manual).


---

**New**:

- Add `temporary_module`, a context manager that uses `create_module`, and automatically removes the temporary module from `sys.modules` when the context exits.

- Add a global postprocessor hook facility. Hooks are called, in the order registered, by `global_postprocess` when the macro expansion of a module is otherwise done. This e.g. allows a macro library to use its own `ASTMarker` subclasses for internal communication between macros, and delete (only) its own markers when done. See `add_postprocessor` and `remove_postprocessor` in `mcpyrate.core`.

**Fixed**:

- Run-time part of `n[]`: upon a parse error, make it clearer in the error message that what was being compiled was an invocation of `n[]`, not the whole source file. (Because these expressions are often one-liners, usually `lineno` will be `1`, which otherwise looks confusing.)

- Fix error message in run-time typecheck of `a` (ast-unquote). Now it mentions correctly what was expected.

- Now `ASTMarker` may contain a statement suite (`list` of AST nodes) as its `body`.
- The debug mode of `mcpyrate.unparse` now renders such bodies correctly.
- `mcpyrate.markers.delete_markers` now deletes such markers correctly, splicing in the `list` of AST nodes where the marker was.


---

**Changed**:

- Update docs: as of `unpythonic` 0.15, it runs on `mcpyrate`, and provides fully functional example dialects based on a whole-module AST transformation.
- The colorizer now injects some styles to `Style` that are missing from `colorama` 0.4.4, particularly `ITALIC`.

**Fixed**:

- Now we pass a filename to `ast.parse` everywhere. This allows e.g. `SyntaxError` during macro-import scanning (in the macro-import dependency graph analyzer), and possible internal errors in the interactive consoles, to report the filename correctly.


---

**New**:

- The unparser now recognizes hygienic captures and destructures them in debug mode. This makes the result much more readable when you unparse an AST that uses a lot of hygienic unquotes.
- To see it in action, use `mcpyrate.debug.step_expansion` macro on [`unpythonic.syntax.tests.test_lazify`](https://github.com/Technologicat/unpythonic/blob/master/unpythonic/syntax/tests/test_lazify.py). See particularly the *HasThon* test; both the autocurry and the lazifier produce many hygienic captures.

Without this helpful destructuring, the macro-expanded code is completely unreadable, but with this, it only exhibits mild symptoms of parenthesitis. For example, this snippet:
python
filename=callsite_filename()

becomes, after autocurry and lazification,
python
filename=$h[Lazy]((lambda: $h[maybe_force_args]($h[force]($h[currycall]),
$h[Lazy]((lambda: $h[force]($h[callsite_filename]))))))

Here each `$h[...]` is a hygienic capture. That's seven captures for this very simple input! Compare this notation to the actual AST representation of, e.g., `$h[Lazy]`:
python
__import__('mcpyrate.quotes', globals(), None, (), 0).quotes.lookup_value(('Lazy',
b'\x80\x04\x95 \x00\x00\x00\x00\x00\x00\x00\x8c\x13unpythonic.lazyutil\x94\x8c\x04Lazy\x94\x93\x94.'))



**Fixed**:

- The importer now reports the source location if destructuring a macro invocation candidate fails.
- Some internal functions, including `mcpyrate.expander.destructure_candidate`, now take a mandatory `filename` kwarg for this purpose.

- Fix detection of globally bound macro invocations (hygienic macro captures) in the helper method `mcpyrate.expander.ismacrocall`.

- Fix syntax analysis for detecting `expr` macro invocations in `mcpyrate.expander.destructure_candidate`. Version 3.3.0 (and only that version) errored out on the AST for `f()[...]` even if `f` was not bound as a macro.


---

**New**:

- Debug hook added to `mcpyrate.core.BaseMacroExpander` to see what the macro expander is doing. The `step_expansion` macro now uses it (which see for usage), but you can also hook your own functions to it.

- Public function `mcpyrate.quotes.is_captured_value` for advanced macrology. This allows your own macros to detect expansions of `q[h[somename]]` in the AST, and grab `somename` (original name, no name mangling) as well as the corresponding value. (There is also `is_captured_macro`, but the use cases of that are much more limited.) Detailed explanation in docstrings for now. Usage examples in the tests for the `quotes` module.

- `mcpyrate.walkers.ASTTransformer` and `ASTVisitor` now have a method `generic_withstate`, to temporarily replace the state when visiting the direct children of the given node. (This is a closer equivalent for `macropy`'s `set_ctx`, sometimes useful for writing custom walkers.)

- Improve documentation on creating magic variables: add another major strategy, and explain both strategies in more detail.


**Changed**:

- `step_expansion` and `stepr` now accept the string `"detailed"` as a macro argument (in addition to the earlier `"dump"` that selects the AST dump renderer).

When `"detailed"` is given, they will report every macro expansion using the debug hook. This facilitates debugging of macros that expand inside-out (using explicit recursion). The definition of *step* remains the same: the `step` counter is incremented whenever the debug stepper gets control back. Just as previously, **inside-out expansion therefore occurs within one step**, but now you can see the subtree of each inner macro invocation just before and after that macro expands.

In block mode `with step_expansion`, one complete step is defined as expanding each statement in the suite by one step.

The macro arguments for `step_expansion` and `stepr` can be passed in any order.


**Fixed**:

- Fix subscript slice handling in unparser for Python 3.9 and later. Now that `ast.Index` and `ast.ExtSlice` are gone, an `ast.Tuple` may appear directly in the slice position, representing multi-dimensional indexing. Such a tuple must be rendered without surrounding parentheses, because the notation `a[1,2:5]` is fine, but `a[(1,2:5)]` is a syntax error. See https://bugs.python.org/issue34822

- Fix bug in quasiquoting of constants: support also `...` (the `Ellipsis` singleton).

- Fix bug in `splice_ast_literals` (a.k.a. run-time part of `q`) that made it crash on `ast.Nonlocal` and `ast.Global` nodes.

- Fix bug in type preservation of empty list in `ASTTransformer`.

- Fix bug in copy support of `ASTMarker` objects. Now it is possible to deepcopy ASTs that contain markers.

- Fix bug that caused the `mcpyrate.debug.show_bindings` macro or the REPL consoles to crash upon a specific kind of broken imports in user code. (E.g. accidentally binding a macro name to a module object instead of a function object.)

- Fix bug failing to honor possible overrides to `sys.stderr` in various debug-printing facilities. Always `import sys` and refer to `sys.stderr` to resolve the current value, never `from sys import stderr`.

- Up to Python 3.8, items in the decorator list cannot be subscripted, so decorator macros could not take macro arguments. In 3.9 this has been fixed, as implied by [the grammar](https://docs.python.org/3/reference/grammar.html). To work around this issue in earlier supported Python versions (3.6, 3.7, 3.8), we now support parentheses as an alternative syntax for passing macro arguments, like in `macropy`. Note that macro arguments must in any case be passed positionally! (Reasons documented in the comments of `mcpyrate.expander`.)


---