Lark-parser

Announcements

- This is likely to be the last major release that supports Python 2 !

We are now working on a **Python3.6+ only** v1.0 branch, which will soon become the default. See the work in progress: https://github.com/lark-parser/lark/pull/925

- We also have a new online IDE! Check it out here: https://lark-parser.github.io/ide

- Lark can now generate standalone Javascript parsers! Check it out here: https://github.com/lark-parser/Lark.js (still in beta)

Changes

- Using rule repeat (`~` syntax) is now much much faster for large numbers, thanks to MegaIng

- Bugfix for the `propagate_positions` option. Added option value `propagate_positions='ignore_ws'`.

- Fixed reconstructor for when `keep_all_tokens=True`

- Added `merge_transformers` (Thanks Robin!)

- Many minor bugfixes, and improvements to code and docs

Cache
- Lark now tracks changes in imported grammars (`%import`), and updates the cache if necessary
- Added support for atomicwrites, for multiprocess caching and crash recovery

InteractiveParser
- Now an official interface (renamed from Puppet)
- Added `Lark.parse_interactive()` for starting the parser in interactive mode

Other

- Added `ast_utils`, to assist in tranforming `lark.Tree` into a customized AST.

- Better docs

- Bugfixes

Notification: Support for Python 2 is ending

In the near future, Lark will drop support for Python 2. We will continue to develop for Python 3.6+ only, which will simplify the code and ease development.

Old releases (including this one) will still work, of course, and should be stable enough to accompany the remaining Python 2 users into the sunset.

If you have any objections, feel free to voice them here: https://github.com/lark-parser/lark/discussions/874


Thanks for everyone who helped make Lark better!

New Features:

- Better grammar re-use with the `%override` and `%extend` statements, which allow to rewrite and extend imported rules and tokens, similarly to class inheritance. (See this example: https://github.com/lark-parser/lark/blob/master/examples/advanced/extend_python.py)

Improvements

- Indenter now throws DedentError instead of AssertionError

- Improved the Python3 grammar, now works with reconstructor. (See this example: https://github.com/lark-parser/lark/blob/master/examples/advanced/reconstruct_python.py)

- Lots of refactoring for a better tomorrow.
- rule/terminals names can now be in unicode. (thanks julienmalard)
- Better errors.
- Better type hints.
- `lark.lark` is now part of the standard library.
- Earley:
- Now works with match_examples()
- Now supports a custom lexer
- Better handling of ignored terminals
- Faster forest visiting, and a few edge-case bugfixes (thanks chanicpanic)

Other

- Lark now accepts funding as a member of Github Sponsors! See here: https://github.com/sponsors/lark-parser

- LALR parser

- The LALR parser now supports priority in rules, as a way to resolve collision errors

- Improvements to the standalone tool, including more command-line options, like optional compression for the json data.

- Improvements to the puppet error handling interface

- Better error reporting on LALR collisions

- Bugfixes in Earley

**Misc**

- Added support for syntax highlighting in Atom

- Fixes and improvements for the `cache` option. `cache=True` now uses a temporary directory instead of working directory.

- Lark can now be imported directly from a zip (See: ed5c8ec51c4c6e8bd0ac80caff6afcb90a97d218)

- Added more terminals to the grammar library (available for %import).

- Nearley tools now supports case insensitive strings

- Deprecated some interfaces

- Improvements to docs, stubs, and various bugfixes

Thanks to MegaIng for helping with Lark's maintenance, and to ldbo, chanicpanic, michael-k, ThatXliner and everyone else for their help and contributions.

- Complete overhaul of documentation. Now using sphinx to generate API docs from docstrings. (commit 0664cbd3d3c19e321cae8df044839e7baf7135af. Thank you chsasank !)

- Many improvements and additions to documentation

- New and friendlier Earley SPPF interface! (commit 555b268eb26bcbfce64991ea7517338dee85a840. Thank you chanicpanic !)

- Added the `ambiguity='forest'` option. Added `ForestTransformer` and `TreeForestTranformer`.

- Various Bugfixes to improve the handling of ambiguous results.

- Read the docs here: https://lark-parser.readthedocs.io/en/latest/forest.html

- New Vim syntax highlighting for Lark (https://github.com/lark-parser/vim-lark-syntax Thank you omega16 !)

- Lark now loads faster from cache (commit 7dc00179e63efa6e98d688bfba3265d382db79c4)

- Terminals can now be composed of regexps and strings with different flags, if using Python 3.6+ (commit e6fc3c9b00306e3a8661210fcc93bf50479ee229)

- Added support for parsing byte-strings, with the `use_bytes` flag (commit 9ee8428f3f6ad285ad93e2b62ec47d33fff54768).

- `UnexpectedToken` exception now has the `accepts` attribute, which contains a list of terminals that would be accepted by the parser instead (in addition to the `expects` attribute, which is guided by the lexer and may include terminals that won't be accepted by the parser) (commit a7bcd0bc2d3cb96030d9e77523c0007e8034ce49)

- Allow multiline regexes with the `x` flag (commit 9923987e94547ded8a17d7a03840c4cebce39188)

- Lark no longer uses the default logger. Instead uses `lark.LOGGER`. (commit 7010f96825b5fbac79522d1b30689065df53dc8c)

- Lark now notifies on unused terminals/rules through `logging.debug`.

- Standalone generator now creates smaller files (without comments and docstrings). Also undergone various fixes. (commit bf2d9bf7b16cddb39f2e0ea3cefecc8de5269e2c)

- Wheel distribution due to (somewhat) popular demand.

- Lots of small bugfixes and improvements!

Many thanks to MegaIng for his continued work on many of these new features and fixes, and to everyone else who contributed to Lark and helped make it even better.