Lark-parser

Lark transformers can now visit tokens as well. Use like this:

python
class MyTransformer(Transformer):
def TOKEN1(self, tok):
return tok.upper()

def rule_as_usual(self, children):
return children

MyTransformer(visit_tokens=True).transform(tree)


Fixed a few regressions that I accidentally added to 0.7.4

- Fixed long-standing non-determinism and prioritization bugs in Earley.

- Serialize tool now supports multiple start symbols

- `iter_subtrees`, `find_data` and `find_pred` methods are now included in standalone parser

- Bugfixes for the transformer interface, for the custom lexer, for grammar imports, and many more

- Added a new tool called Serialize, that stores Lark's internal state as JSON. That will allow for integration with other languages. I have already started such a project for Julia: https://github.com/erezsh/Lark_Julia (It's working, but still in early stages)

- Minor bugfix regarding line-counting and the \s regex

New features:

- Lark now allows you to specify the start symbol when calling `Lark.parse()` (requires pre-declaration of all possible start states, see the `start` option)

- Negative priority now allows in rules and terminals (default value is still 1, may change in 0.8)

Also includes many minor bugfixes, optimizations, and improvements to documentation

- Lark can now serialize its parsers, resulting in simplified stand-alone code.

- Bugfix for v_args (Issue 350)

- Improvements and bugfixes for importing rules from grammar files

- Performance improvement for the reconstructor feature

This new version includes a brand new Earley implementation, with support for a Shared Packed Parse Forest (or SPPF), providing much better performance for ambiguous grammars, both in terms of run time and memory consumption.

Users might notice a small degradation in Earley's run-time performance for deterministic grammars. Hopefully future releases will take care of that as well.

Big thanks to night199uk for his great contribution.

Other features worth mentioning (though they exist in the previous release):

- Added support for importing rules between grammars. The import mechanism is namespace-aware.

- Added the `maybe_placeholders` option, which causes optionals of the form `[expr]` to return `None` when not matched, instead of just not appearing. (optionals of the form `expr?` maintain the previous behavior of not appearing unless matched)

- Plenty of bugfixes, better errors, and better docs