Hfst

----------------------------

* Numerous improvements in pmatch processing:

* hfst-pmatch2fst now implements eg include"filename.txt" as a preprocessing stage

* hfst-pmatch2fst now implements Explode() and Implode() in full generality

* hfst-pmatch now allows printing weights (in location mode, with --locate)

* hfst.PmatchContainer.locate offered via python bindings

* bugfixes and improvements hfst.PmatchContainer.tokenize

* Add tool hfst-eliminate-flags.

* Allow both escaped and unescaped zeros in multicharacters in all lexc lexicon entries (fixes issue 274).

* Also reweight state that is both initial and final in hfst-reweight (fixes issue 368).

* Do not let distributed files depend on configuration. Include man files in dist.

* Print warnings and error messages also in earlier stages of twol compilation.

* Add option --execute-and-quit (-E) to hfst-xfst.

----------------------------

* Numerous improvements to pmatching and tokenization:

* pmatch now supports the default symbol

* pmatch now supports reading word embeddings in the binary format

* improvements to pmatch runtime context handling: a bug affecting expression-initial contexts is fixed, and pmatch now supports Ins() arcs inside RC() and NRC() contexts

* in pmatch, bugs affecting multiple Ins() arcs, in particular nested ones, are fixed

* Implement variable 'retokenize' in hfst-xfst

----------------------------

* Numerous improvements to pmatching and tokenization:

* [].with(X = Y) feature in pmatch This provides support in the pmatch2fst compiler to define "global flags".

* Add a variable "xerox-composition", default to "on".

* Consider list symbols (L..." and "X...") to be special.

* Fix runtime handling of contexts and compilation of negative contexts.

* Make Like() and Unlike() much faster by not sorting the whole vocabulary and only calculating each comparison key once.

* Keep track of weights along context checking paths and unify weight handling.

* In blankline-separated mode, keep blanklines in output too.

* Round weight to zero decimals, non-sci.

* Make hfst-tokenise usable as a lib; include simple string-to-string function.

* Use libreadline in hfst-pmatch when available.

* Cleanup remainder of the pmatch test suite, all the tests now pass.

* Python interface:

* Support reading several twolc files.

* Add functions 'compose' and 'cross_product' that take a list of transducers.

* Allow empty string as input for hfst.fst and hfst.fsa and interpret it as epsilon.

* Perform fsmbook tests also via python API.

* Add option --local-hfst to setup.py.

* Include pre-swig-generated wrappers to pypi source distribution.

* Compilation:

* Use by default c++11 unordered_map and unordered_set, unless otherwise specified.

* Add an option --without-c++11 (defaults to 'no') to compile hfst without c++11 support.

* Require libc++ and osx version >= 10.7 with clang.

* New functions and options:

* Add function HfstBasicTransducer::remove_final_weight.

* Add function HfstTransducer::negate() for automata.

* Add option --restricted-mode (-R) to hfst-xfst.

* Flag diacritics: support getting a list of operations involved with a particular feature.

* Allow creating HfstInputStream's from std::istream's.

* Fix issues 341 and 353, make workarounds for issue 358.

----------------------------

* Changes to configure:

* Disable lexc and foma wrappers as well as hfst-train-tagger tool unless explicitly requested

* Enable hfst-calculate and hfst-xfst by default

* Add experimental options --with-openfst-log=lean and --with-sfst=lean which support a limited nuber of operations for these types (reading, writing, converting between types and deleting)

* Require at least automake version 1.12 unless compiling from pre-yacc-generated sources

* Implement hfst-twolc as a single program instead of a script. Get rid of hfst-twolc-system and hfst-twolc-loc tools.

* Improve pmatch compilation and error handling mechanisms

* Improve hfst-tokenize tool

* Add transliterate output mode (--transliterate) to hfst-proc

* Changes to python interface:

* Support twolc and sfst compilation

* Improve HfstBasicTransducer iteration mechanism

* Add experimental support for python version 2

* Support apply up and apply down commands in function start_xfst

----------------------------

* Fix flag elimination bug (reported in issue 342)

* Do not allow unescaped dots in regular expressions

* Improvements to pmatch and tokenization tools:

* Search for included files under scriptdir, not working dir

* Add experimental two-vector model for word sense

* Handle Apertium-style superblanks in --giella-cg format

* Rename the PyPI package to 'hfst' (currently available at testpypi.python.org)

* Update foma back-end

----------------------------

* fixes to memory leaks and efficiency

* fixes to numerous warnings

* changes in Python bindings:

* rename hfst.rules into hfst.sfst_rules

* get rid of hfst.types and offer implementation types in class ImplementationType

* add Xerox-type rules in module hfst.xerox_rules

* improve documentation

* tentatively add partial support for pypi installation

* improvements to pmatch tools and hfst-proc