Hfst

---------------------------

* this is a bugfix release

* conversion to hfst-optimized-lookup format was fundamentally broken in the
original release; this should provide a fix and considerable speedup

* compose-intersect memory use is much improved

-------------------------

* all new API in object oriented form

* brand new foma backend for unweighted transducer handling

* the FST libraries used for automata handling are now dynamically linked to
HFST

* this requires end users to install needed backends themselves

* hfst-lexc and hfst-xfst based on foma's compiler

* xfst is now also fully supported

* the obsolete command line clients for HFST-based compilation of lexc and
xfst scripts are hidden in hfst-lexc2fst and hfst-xfst2fst

* preliminary support for unknown character classes

* optimized-lookup backend supported in library

* dropped special tools for optimised handling since optimised lookup is
included in library: hfst-optimised-lookup -> hfst-lookup;
hfst-lookup-optimize -> hfst-fst2fst

* hfst-proc tool to emulate lt-proc, cg-proc

* Slightly improved string and text formats to support weights uniformly and
parsing tabs and spaces as automata arcs: see `wiki page for text and string
formats
<https://kitwiki.csc.fi/twiki/bin/view/KitWiki/HfstStringAndTextFormats>`

* better error handling using exceptions

* The argument handling order of binary tools changed: stdin is first if not
specified

* The automata are now packed in HFST automaton container format (akin to AVI
for videos); this supports arbitrary metadata and underlying FST formats

* the symbol handling is more implicit; no more symbol table arguments for tools
or hfst-symbols command

* converted to new autotools: silent rules and color tests

* Hfst's internal automata format for fallback on some unsupported operations
and for conversions

3d3e70be Fixed a bug in compiling rules with word boundaries and added test45.
7ff82000 Don't use sed in scripts since it's broken on MacOS
cb7b1ee2 Fixed bug in compose_intersect which discarded the second rule systematically.
948b3721 _SPACE_, _TAB_ and _COLON_ added to HfstStrings2FstTokenizer.cc
7ad47d77 hfst-lookup now uses HfstFst2StringsTokenizer to tokenize the input strings.
9ab65922 Don't convert same stack more than once
b9dbb3ab fisx some warnings
fa5dd096 workaround mac''s missing strndup
6870c098 Allow pair substitutions...
01d746d8 Fix test for Mac OS
677dddef Replace tabs with random number of spaces
0990ea4a More tests limited to available formats
8ee702b9 Only test available automata formats
96af3b2d Test all string formats in strings2fst
834cf29e Fixed hfst-string2fst parsing of input strings and reading of multichar symbols.
e1880818 hfst-strings2fst should now be able to handle all kind of string formats.
ecc96f2b is no longer a special symbol, because it is not a special symbol in xerox twolc.
2ca9026a Fixed print in left arrow rule conflict warnings.
3d41fc23 Fixed word boundaries in compose-intersect, again...
b4270dd1 Special symbol handling changed in the commandline tools.
0e1bea82 Fixed word boundaries in compose_intersect. When HfstBasicTransducer substitute starts working correctly, this can be done more efficiently.
161d9032 Fixed word boundaries in compose_intersect.
f507cee7 Added printing the conflicting context in left a row rule conflict warnings.
2851f19b Unknown and identity symbols are now correctly handled in composition. AT&T format now allows spaces in symbols if they are ascaped as '_SPACE_'
caaf1098 Tests for ? symbols
a57fcaad Use unknowns and identities in regexp2fst
a6c98345 Documentation added. Exceptions now throw a message.
dfceeeed Modified test so that the script does not have to be changed when new tests are added.
71f1022e Removed a unnecessary newline printed when not reporting left arrow conflicts.
6b311ec8 Added a rule for compiling a local version of hfst-twolc used in tests.
d08533d7 Added hfst-twolc tests.
e032ace0 Added hfst-twolc tsts.
98a54028 Added hfst-twolc tests.
aacbe751 Cahnged order of arguments so that 'cat fst1 | hfst-binary-tool fst2' will treat fst1 as the first and fst2 as the second argument.
262d19eb Fixed compose-intersect
1bb5dec7 Handle the empty transducer in ->ol
34d35770 Now hfst-compare does not print anything in silent mode. More tests added to test/libhfst.
ea109120 Fixed definitions
232b892a Fixed d:0 where d is a diacritic.
58836d2a Fixed symbol offset calculation in certain flag diacritic-concerning cases
d2a766d4 Fixed problem with diacritic pair syntax. Now d d: and d:0 are equivalent for the diacritic d and d:x will give a warning.
a5505dc6 Remove debugging strings
0590acdc Properly copy flag state vector on stack
25a5b9ed Fixed a bug with definitions
32e01ece Add tests for UTF-8 and latin-1 encoded strings
ef669e3d follow print-epsilons in all formats
9331c511 format options like other tools have them
5b8b2b15 More documentation added.
6197f2fe More careful quotation and informative error messages for foma scripts
5d6a2ffa Use "HfstTransducer.hh" instead of <hfst/HfstTransducer.hh> for in tree builds
49db47a5 hfst-strings2fst fixed
6bf1608e Now hfst-strings2fst should handle escaping correctly.
2825f8da A bug in hfst-strings2fst fixed: now the backslash is reserved for escaping characters. Now the library also throws more informative exceptions.
54aaa9ec Fixed operation retrieval
1a0485aa Fixed problem wrt subtracting from an end() iterator of an empty state_placeholder index placeholder map
85d0602c Removed some more debug prints.
fe95ee70 Removed some debugging prints
28844b5b Fixed bug in defining lista using other lists and another bug which prevented variable rules unless there was a Definitions section.
613d39f6 Fixed diacritics handling.
86c83356 Fixed output file handling.
103d9b23 Gave up on making }, ] and ) literal if they are not preceeded by opening {, [ and [, since this is too difficult in flex when brackets get nested. Fixed commandline argument handling properly and separated { and } from [ and ] i.e. they get their own bison symbols.
52ebbb7c Fixed handling of commandline paramters so that the input filename can be given as a free argument without -i or --input.
d630caf9 Fixed syntax so that a lonely }, ] or ) is not interpreted as abracket, parenthesis and so on. Fixed numbers so that they need not be quoted unless preceeded by ^
ade6ed80 nore quotation and script guessing
60c80232 Added hfst-compose-intersect.cc
2b7a964f Changes needed for libhfst/src/implementations/compose_intersect
5718b7b6 Added fiels needed in compose_intersect
803e57e4 configure.ac changed because hfst-twolc was added.
51b9cd2d Removed file hfst-twolc/.txt_fst which was added by mistake.
cfa3c5ce Removed geerated dir hfst-twolc/src/.deps which was added by mistake
38ec61e1 Added hfst-twolc.
e195be67 Miscellaneous packing tricks & fixed a typo
78152220 Correction to flag indexing
51cdc7ac Corrected alphabet ordering (eps-input-flag-other)
a07a6e43 Small changes done in documentation.
f8658bb0 More refactoring
20e8d361 Miscellaneous improvements and refactoring
4e9475fe Fixed bug wrt finding the largest used index
859a1b27 missing test file added
b4ecec5d Parametrized packing aggression. Packing is still a little slow for decent levels of aggression. Consider adding a mechanism to ask conversion to be maximally aggressive (eg. for binary distribution).
fcaea641 Redo flag handling in ol-conversion
dfb26903 A bug in hfst-lookup fixed: reading input strings in spaced format now works.
7856fc99 Fix stupid mistake in previous fix to optimized-lookup internals
2f321110 More coding style conformism
bc68f4ce Coding style conformism
7d828ce8 Fixes to optimized-lookup internals and conversion
d53894f2 Exception classes refined
844d8816 All exceptions changed to HFST_THROW(HfstException).
66ebc7df Renamed TWROW HFST_THROW in HfstExceptionDefs.h
661b615d Added some macro definitions, which make it easy to declare, define and throw informative exceptions.
829953ab Free memory earlier and do some less work in ol-conversion
3c709019 The order of weight and string/string pair changed in HfstOneLevelPath and HfstTwoLevelPath. Now the elements of a set are sorted in ascending weight order.
44f919c2 HfstLookupPath(s) is now replaced with HfstOneLevelPath(s)
ae4f393d Flag diacritics should now be handled right in functions lookup and extract_strings
95eca273 HfstTransducer::lookup now takes a StringVector as argument and HfstTransducer::extract_strings stores its results in an 'HfstTwoLevelPaths'
38c5e2b2 Now coding-style.sh should pass.
7470a480 Overlong lines separated on several lines.
525715b8 More more more OFST->OPENFST renaming
bd8523a7 More more OFST->OPENFST renaming
1a562676 more OFST -> OPENFST name renaming
645f7af9 HfstTransducer::read_in_att_format is now protected
4b37099c A bug in tests fixed.
b1067964 test.cc renamed to test_rules.cc and only tests for functions in namespace hfst::rules preserved
708daa6a TROPICAL_OFST_TYPE and LOG_OFST_TYPE changed to TROPICAL_OPENFST_TYPE and LOG_OPENFST_TYPE.
4a1fd2ff Fixed a bug in reading binary SFST and FOMA transducers.
80e53b48 update requirements and reorganise
44a7d7ab Fixed bug in reading and writing transducer binary headers.
d9ac6ecf AT&T format now accepts all whitespace characters as field separators.
6fd24d8d Added a const-version of HfstTransitionGraph::operator[], so that it is possible to access the transitions of a const HfstTransitionGraph.
3cae2e6a Added appropriate type checks in a HfstTransducer constructor - probably many more needed...
af09e6fb Fixed typo and clarified help message
ce45fb53 Represent epsilon as blank string when returning representations from ol-library
13f7061f More tests added. Substitute bugs fixed in TropicalWeightTransducer.
5632573b New tests added. Substitution functions cause a segfault in TropicalWeightTransducer, debugging under way.
2c7c03e8 More tests added.
91ab1092 Fixed some problems (due to epsilon symbol) in SfstTransducer functions that take a String or StringPair as argument. FomaTransducer::is_cyclic now works correctly, function 'fsm_topsort' must be called before 'is_loop_free'.
2ea66592 More tests added.
ebfd3f97 File test_transducer_functions fixed.
47066b22 Extra debugging messages removed.
e48223df Test for tokenizer added.
8ff3b25a FomaTransducer::define_transducer(char*) now calls fsm_count(struct *fsm) on its return value. In this way the number of fsm lines is correct when the transducer is written to file.
8485a6dd More tests added.
10901e1d git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk925 941e2c2b-deac-454f-805a-451daa25f33c
a68b3e3c git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk924 941e2c2b-deac-454f-805a-451daa25f33c
0df13e98 File test_constructors.cc added
e1e8ffef Documentation added.
7e81b7b8 Documentation updated. Tool hfst-name added.
ab09facb Now conversion from OpenFst and SFST to HFST handles special symbols correctly. How special symbols should be handled in the reverse case is still an open question.
aaa59d31 remove deprecated conversions and unused parameters
8cbe4a23 ol-basic -conversion
b3a201a2 Function HfstTransducer::insert_freely(const HfstTransducer &tr) added.
63eb0309 Now the output symbol table of a tropical or log HFST transducer is assigned a copy of the input symbol table when the transducer is written. This should make conversions between OpenFst and HFST transducers easier.
473157e5 Fixed two typedefs in trunk/hfst3/tools/src/hfst-summarize.cc
c8c1af8e Fixed an include in trunk/hfst3/libhfst/src/implementations/ConvertTransducerFormat.h and a typedef in trunk/hfst3/libhfst/src/implementations/convert.h
25fc8e1a Fix verbose messages about HFST3 headers
e856b97a epsilon tests
0e91640d hfst-lookup now prints the results in the FILE requested instead of stderr
758a9565 hfst-lookup has two options more. It is possible to print results in pairstring format (not implemented for optimized lookup format) and defined how the epsilon is printed.
76acf98f extract_strings and WeightedPath changed so that it is possible to include a StringPairVector representation of the extracted paths
d17c308a implement flags and epsilons and xfst options print-space print-flag obey-flags etc.
290b8612 Avoid testing missing backends if possible
843e8d61 Documentation added. Thrown exception types changed for some functions.
ed4c9afc Documentation improved and added. Keyword 'const' added to functions and their arguments when appropriate.
532d9e90 Documentation added.
026a0956 Documentation changed.
a76fccfa HfstFileException changed to HfstStreamException
740219f4 Missing flag diacritics now yield a warning in hfst-compose, hfst-conjunct and hfst-subtract, unless freely inserted by option -F.
fbd97a15 One more portability fix... could be the last one!
200fae13 Yet another without-openfst portability fix
da6dfc85 Another without-openfst portability fix
beae8209 Another without-openfst portability fix
b8f670a9 Added using-directives to hfst-lookup.cc - apparently they don't get included from somewhere else when not HAVE_OPENFST, so this is necessary for portability
e9a2fa57 More optimized-lookup conversion fixes... produces correct results at least for trivial cases now.
37717b2a Various bits of format correctness for optimized-lookup conversion (lookup still doesn't give correct results)
1e51ff91 Documentation added.
c929f0b8 HfstAlphabet, HfstBasic and HfstUtf8 cc and h files moved to tools/src. Some functions in HfstTransducer decalred as protected.
3e5198b9 if HAVE_HFST_OL perhaps not working correctly - in any case HFST_OL-writing is now enabled, although the results still don't appear to be quite correct
70441e4d Oops, forgot to remove some debugging messages
c930115b Fixed hanging hfst-ol conversion, still doesn't get written
4b3123b6 Ol-conversion (untested as yet)
7f0965a4 Added parsers/xre_utils.h to noist_HEADERS in tools/src/Makefile.am
52ba7b8d Added parsers/XreCompiler.h to noinst_HEADERS in tools/src/Makefile.am
0382dde3 Documentation added.
32100129 Documentation added.
94820cdd Intersecting composition functionalities removed, they will be implemented using HfstBasicTransducer format in the future.
90253d69 Updated configure.ac to reflect dropping of tools/src/parsers/Makefile.am
081c986f Deleted unneeded Makefile.am in tools/src/parsers
aa3fcaad Substituting unknowns and identities now works correctly in composition.
3e96e8ff A bug in reading transducers in AT&T format fixed. Also an alphabet bug in converting HfstBasicTransducers into SFST transducers fixed.
7c9ed801 Corrected documentation link.
7bd61dfb Documentation added.
b03365dd Redundant code removed from class HfstTransitionGraph.
ee7689ac HfstNet renamed to HfstTransitionGraph, HfstFsm to HfstBasicTransducer and HfstArc to HfstBasicTransition
9c3b6a60 extra_dist all scripts
b930429f fix VPATH _build_
324cd91d Substitute functions of HfstNet updated.
366b7839 HfstCompiler.h in noinst_HEADERS
d545f778 Added items to noinst_HEADERS in tools/src/parsers/Makefile.am, evidently needed for building
150715f9 Remove intermediate files and be selectively verbose to avoid confusion
e2e9e79a YAUH: HfstUtf8.h
4c75eabe YAUH: HfstBasic.h
f0e1f0c4 Another unshipped header (HfstGrammar.h)
c9d913a2 oops
b258f1f5 HAVE_OPENFST guards
ab00fb47 A bug in transducer conversion functions fixed.
ab4c8404 Missing \ in another header shipping directive in libhfst/src/implementations/Makefile.am
8bb83a4a Another missing header, TropicalWeightComposeIntersect.h included for shipping (this is probably temporary and the header shouldn't be needed anymore, but for now building fails without it on clean machines)
558f4f35 The alpha testing instructions
cc748ccf skip all tests that might fail
1ac0e16a * some new autoconfiscate features (silent make!) * misc. stylistic changes
e8b5ac08 Made HfstNet::get_final_weight(...) a const member function.
eafc2e8c Added missing headers to Makefile
24277857 changed void HfstNet::add_state(HfstState s) so that it returns s and added HfstState HfstNet::add_state(void), which adds a new state.
51d9a4af HfstNet.cc added
6468e7fa Conversions between transducer formats updated so that they use HfstFsm
236c20d5 Fix the missing includes and linkage
5a74f1f1 Remove hfst's lexc, xfst and xre parsers and use foma's instead of reinventing wheel and doubling update burden
f569bf39 New transducer conversion functions added.
b1b2c7c2 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk841 941e2c2b-deac-454f-805a-451daa25f33c
569ec24e A bug in FomaTransducer::define_transducer fixed.
bb4aa6b2 A bug fixed in HfstNet::substitute(StringPair, HfstNet)
7b4516ee More functions added to HfstNet
c80a3412 More functions added to HfstNet
59c60c3c Use local compare for all tests
4babf628 don't fclose infile too early
f4ecf799 * Separate bug report from further info by newline * remove old and deprecated tools
7db67bb3 * intersect has been called conjunct since forever * move external links in SEE ALSO
c49c6c73 enforce coding style in make check :-)
a5774a96 File new-ConvertTransducerFormat.cc added, it will soon replace ConvertTransducerFormat.cc
3c754389 testing results of parameter vcombinations
1c76cc0e add hfst-lexc script
1fe743b0 Update xfst/foma scripts
9259bbe9 Added a substitute function to class HfstNet.
ea060127 HfstNet.h has now a function write_in_att_format
a0e38e29 HfstNet.h updated and tested.
692777ca Added file HfstNet.h that will soon replace file HfstInternalTransducer.h.
8bd82472 Fix internal tests
a988bf90 HfstInternalTransducer_ updated.
d511822b A new HfstInternalTransducer class HfstInternalTransducer_ added.
5105bb2f class HfstTransition changed to struct HfstTransition
de7d5ce4 A bug in converting empty transducers between internal and OpenFst's formats fixed.
b2b1ee87 use if's on backend specific code
75bc715f LogFstTrieFunctions .cc and .h removed and their functions moved to LogWeightTransducer .cc and .h and copied also to TropicalWeightTransducer .cc and .h
e5cacc1a Files renamed and deleted.
19e3b71d Including headers should now work correctly. Doxygen documentation also added.
871e1e0f The minimal lexc compiler functionality tests and minor fixes to compilation formula
4d00374a implementations unit tests
4233b9d5 Add unit tests for extensions
e7213c65 don't test default input stream since it requires stdin
135c50f5 EXIT_* needs cstdlib on newere gcc's/glibc's
f0084381 main interface unit tests 3/6
7923d45e It's a verbose test suite
3fdf3eaf hfst-strings2fst now tokenizes multicharacter symbols
4280623c A bug in hfst-strings2fst with (not pairstrings && not spaces) fixed.
dc4b60bc hfst-strings2fst now uses the internal trie transducer format and converts to the format requested before writing the output. This should make hfst-strings2fst faster.
178b960b Documentation added.
39f65532 Added a simple example transducer library, MFSTL, that can be included by command './configure --with-mfstl=true'
37fa1735 An implementation type FOO_TYPE added for testing, by default it is not included in HFST.
779e27e3 Fix lookup_printf and test it
91cbdcaa Reading hfst version 2 transducers whose implementation type is OpenFst should work now.
72481314 Documentation added. A bug in hfst version 2 transducer handling detected, fixing started.
8a0f6926 Full-ish HfstTransducer unit test
f65f2285 A bug in hfst-strings2fst fixed, now the newline character is discarded before converting the weight string into a float. More Doxygen documentation also added.
76ebbd1a Lookup implemented for optimized-lookup API with new style headers, though hfst-lookup doesn't print anything due to commented-out printing function (it seems to break when I uncomment it..?)
606ddf69 This could be unit test
8887df69 hfst-lookup should now work correctly for transducers having epsilon input cycles. Also added some documentation.
da33a15a fix more of mac os x getline substitute
b88d9ff5 skip hfst-proc dir for empty input tst
b41e5743 Revert man bootstrapping stuff and add stubs for rest of the tools
539ef76f add set and use zu for size_t prints
100123f8 hfst-lookup fixed, still allows too many cycles although limits the results to a finite number
0aeebd28 A bug in hfst-lexc-compiler fixed, still some errors in help2man and tool tests.
1e978756 File added.
b185125e Outputs of help commands of commandline tools fixed.
12ff3ef0 A bug in HfstTransducer(StringPairVector&, type==FOMA_TYPE) fixed.
edbb55d3 A version of infinitely_ambiguous (necessary for at least lookup)
d7f8ba0e Removed conditional compilation guards from non-template lookup-performing function (which way do we want it to be?)
d9efbeee enable lexc, missing ) added
64c686b8 Some additional HFST_OL functionality
75934b3a Update man pages for release :-)
2dc49a31 Tests added.
209fd043 Some fixes to make the right names defined down the line, plus a first attempt at the lookup API
39692d3c Lookup for hfst_ol in hfst3
4887fe93 More documentation added.
07b48388 Files documented, a skeleton file added for adding a new backend implementation
28316461 Files renamed
cc616921 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk769 941e2c2b-deac-454f-805a-451daa25f33c
185380b1 Now libhfst/src/implementations contains files that modify backend transducers directly and libhfst/src other files
a98715af Files moved
636599b1 Files moved
a373e2db Set executable bit + line endings.
fcf17ac0 Fake hfst-xfst using foma-wrapper hacks
21581874 HfstCompiler.cc and HfstCompiler.h added to tools/src/
485406cb Files moved.
b7d01a85 hfst-lookup updated, handling flag diacritics and multiple paths with the same weights not yet implemented
bccb155d hfst-lookup no works on unweighted acyclic transducers
565d9908 set type after exiting error for mac compiler
4cba1aa0 Fix -o handling
a62adc65 hfst-lookup implementation continued, works not yet
edd084b5 beginning to implement hfst-lookup
cb1cd036 Recover from make-breaking stupid mistake
2b402e56 Standardization of HfstOlTransducer's iostreams
b10fc116 just testing that everything works
d20527d0 Clean up few minor details
d1968d9b Documentation added.
0fd33bd7 More documentation added, exceptions divided into subclasses.
f55820f5 Documentation added.
bac207d2 Catch stuff to provide meaningful error messages, provide better error replacement
7bec3d4b Small changes done, documentation added.
ddaf962a Lookup and related functions
94bb62d1 Update string_to_format and some polishing
3067f9fc A bug in reading transducers with old header format fixed.
8bde7d58 harmonization function optimized
29c3abc2 don't use bsd err functions
19df4c88 return paths even if !HAVE_SFST
cba810b1 using std::map, have SFST stuff ifdef'd
218faa62 A name field added to hfst header. Doxygen documentation updated.
7a3b4731 HfstInputStream changed
c7509d0f Missing restriction rule operators added.
fd80271e A bug in HfstRules fixed. FEXPORT must be added in front of foma functions 'sigma_create', 'xxmalloc' and 'sigma_max' in file fomalibconf.h until new version of foma is released.
1a306b54 A bug in HFST header handling fixed. Todo: update HFST code to work with the newest version of SFST.
42e25943 Fix lexc compiler
8739b075 Detail GNU bison version and feature requirements
801ad208 Use obsolete syntax for name-prefix to support slightly older bisons
2d9b05b8 More checks for backend installation sanity§
da681d11 Make compiler tools dependent on relevant features in library
a6f4b681 Also include current foma diffs
322b7f80 Function open() removed from HfstInputStream and HfstOutputStream. HFST transducer header structure changed.
5d197d0d Added MacOS X notes.
6532861d Remove bundled foma and SFST
bf5053ed Mergefail→delete duplicate lines
cba481f2 Add preliminary xfst compiler for offline compilation of static files
ea740e3c Add autogen.sh for problematic oses and systems
2ce19e80 Document library backend requirements
d911ae35 add replacement strndup and error_at_line for mac again
dbcfbf10 commandline tool hfst-compose-intersect and test added
45290c5d more tests added to tools, SFST_TYPE transducer header now has an optional field that tells if the transducer is minimal
0db6a0b7 more tests added to test/tools and small bugs fixed in commandline programs
ce7f006a HfstInputStream and HfstOutputStream now work correctly with tropical and log weight transducers if their file argument is an empty string
b36fd197 Alphabet definition in HfstCompiler is now not done through internal format when using SfstTransducers. This optimization makes hfst-calculate slightly faster.
aaa2eae0 tropical transducer harmonization code optimized
5421e3d9 Moved the endif to the actual correct place...
3a950b76 Moved endif to correct place. Suggestion: when there's a lot of distance between if and endif, consider placing a comment after endif to indicate what it's ending (emacs can't figure it out).
8ab00a76 Removed obsolete definitions
15369a72 transducer harmonization code optimized
bf6dfa1f git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk692 941e2c2b-deac-454f-805a-451daa25f33c
7eb10ec4 missing files added
3d244cbe git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk690 941e2c2b-deac-454f-805a-451daa25f33c
3bc47057 HfstCompiler's code optimized
0a590ffb git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk685 941e2c2b-deac-454f-805a-451daa25f33c
dfd51435 replace_in_context's code optimized
0ba83666 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk683 941e2c2b-deac-454f-805a-451daa25f33c
2cad021b git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk682 941e2c2b-deac-454f-805a-451daa25f33c
e1f264b4 Class 'HfstTrie' added for quick disjunction of words in hfst-calculate
41328b3e reading words from a file should now be faster in hfst-calculate
c973556d git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk679 941e2c2b-deac-454f-805a-451daa25f33c
c075ba73 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk678 941e2c2b-deac-454f-805a-451daa25f33c
256cd2df a bug in hfst-calculate's 'make_mapping' fixed
f46a184f now hfst-calculate does not unnecessarily handle unknown symbols in composition
f2a53597 A bug in alphabet definition in hfst-calculate fixed. It seems that handling unknown and identity symbols in composition is very slow for some reason.
d194e80e A bug in HfstCompiler::complement_range fixed
a4899508 HfstCompiler::new_transducer is now faster
e74f90f7 some bugs in command line programs fixed
d78c45a1 some bugs fixed in tests and hfst library
2a64259d 4cats.txt now contains a transducer accepting 4 consecutive 'cat's, and 4_cats.txt contains 4 consecutive transducers that each accept 'cat'.
acd96ec8 a bug in function 'harmonize' fixed
553674c8 Commit missing inputNamed
d9f95515 hfst now compiles with all combinations of available backend libraries
c5e09736 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk661 941e2c2b-deac-454f-805a-451daa25f33c
68b326b9 Make --without-foma work
71d08c49 optimized-lookup bigfix wrt weights in flagged transducers
41a235e6 Function HfstTransducer::push_weights added. Foma is now dynamically linked to hfst, option -fvisibility must not be used when compiling foma, because then some of foma's functions will not be visible to hfst.
fbe29693 Tentatively implement HFST 2 format
9d4aeed6 move things around
917827d7 Add space separated strings2fst; parse - as filename in hfst_fopen
6c693093 hfst now works without making any changes in sfst files
11f54203 Configuring out broken parts
17cd8d38 Add preliminary lexc compiler
ab50a962 HfstMutableTransducer defined in namespace hfst, small bugs fixed in TropicalWeightTransducer
226ae4cc unknown and identity symbols are now handled correctly in composition
c016a92b HfstMutableTransducer replaced with HfstInternalTransducer
792dfb30 Document the XRE compiler
a2d03673 functions that read and write in AT&T format now use HfstInternalFormat
6f0770db some modifications made to HfstInternalTransducer
0d5c47d4 freely_insert and substitute functions now do not depend on any specific library. They must still be tested.
2bf97cb0 some functions added to HfstInternalTransducer
f26af421 invert XRE autoconf logic
1a8a586e missing files added
b2da1a4a conversions between internal transducers and sfst, tropical and foma transducers implemented
9b896d20 Enable XRE parser
058a4093 conversions between SFST and internal transducers added
66cdf07c summarize more statistics
3260a0a4 Wrap backends around autoconf have_
2c140fa3 Enable basic compose-intersect again; no flags, no 's
ef39e843 HFST code updated so that it works with OpenFst version 1.2. New internal transducer format added.
47cdd70e fix signs when wno-sign-compare fails
00e079ba Delete generated files
4590274e Now hfst-calculate does not depend on SFST.
1a56a054 hfst-calculate now does not depend on SFST, function implementations not yet written
ea974ba0 a bug in substitute(StringPair, HfstTransducer) fixed, agreement variables added to hfst-calculate
90d62759 Add some functionality tests for hfst-proc
a884b276 Remove obsolete source files of hfst-proc predecessors
4ebe3484 Fix bug causing final weights not to be included in weight calculations
32abdeca Remove obsolete flag diacritics code from hfst-proc
b5a8fe8e Add license information to hfst-proc and optimized-lookup files
eeedf419 Modify the extract_strings algorithm to avoid the non-standard std::vector::data() for older GCCs
2cad922e Perform initial migration of hfst-proc into its new home in the proper HFST tools directory, while refactoring to share as much code as possible with the optimized-lookup library backend
0fc222ea Add 'exclude' option to hfst-fst2strings
af172ca5 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk616 941e2c2b-deac-454f-805a-451daa25f33c
c4577241 Fixed memory leaks in harmonize
fb3fab5b Added libhfst/src/implementations/TropicalWeightComposeIntersect.h and libhfst/src/implementations/TropicalWeightComposeIntersect.cc
ebef427e Composition can soon handle identity and unknown symbols correctly. Substitute can now also modify symbol pairs through a function pointer.
0207962b Refactor extract_strings to invoke a callback on paths, allowing neat features in hfst-fst2strings
25b747a4 Credit self in AUTHORS
fddd8b22 SFST updated to new test version, some modifications still needed to HFST library so that it will work with the new version
17125a48 error_at_line for parsing errors
5df88b53 Allow empty lines and comment lines starting with except when followed by a tab.
b8da331a Add the basic summarize functionality and the test
5ff805ee rule operators added to hfst-calculate
3618b202 More functionlaity tests
1daa938e Reimplement extract_strings in all backends with cycle-limiting functionality
8ebdabee substitute and test
4c2697a2 Add test for fst2stings
3c0db1e0 more functions added to hfst-calculate
527b7cf5 Ubuntu (dash?) workaround
07c88321 Make string pair implementation work
5a1646a5 Add lexc compiler's old skeleton
768cc079 Reverse how the upper and lower strings are stored during string extraction in the SFST backend
d8228327 Implement string extraction in the optimized-lookup backend
3901e854 Fix a compiler warning about missing a return statement
e55cdabe Remove debug output that should never have been committed
5238401c Fix calculation of several header properties during conversion
7130a070 more functions added to hfst-calculate
35c3e57b Start a document on formats
dbf650c0 test strings2fst and fail
bc5f85db Rollback bogus repeat throw
23f95bcd comment out broken repeats
84d6b262 Fix bug in binary op parameter handling and add test for it
5f3b3172 Invert upper and lower to follow upside-down terminology of HFST
8cb21960 Fix no parameters parsing for unary tools and project
72cb4160 Test basic algebra
8ce84837 functionality test for concatenate
cbecff25 Add text based transducers for functionality tests
124df373 Fix fst2strings to use set_program_name and rest of the basic tools
893186f8 small changes done in functions that are needed for hfst-calculate
8e30ba10 Check handling of empty input file for all tools
11692b29 Default to -Wall -Wextra -Werror for developers
041c510f Convert txt2fst; * use const for global char-* * default stdin, stdout for global FILE*
58dff6eb Add initial support for evaluating and filtering flag diacritics when extracting transducer paths
90f49d95 Add a class for dealing with flag diacritics in string extraction
7f8204c8 Port and enable fst2strings
44154e0d Hook up is_cyclic for the optimized-lookup backend
74dcdfc6 Use warn and err if warnc, errc, error are not available
a425f2b2 Instead of gnulib-tool, handle gnulib manually
aaa4a7f4 Missing file
34e3ecb3 Gnulib weirdness?
075e63cc Convert some simpler command line tools to HFST3 (preliminary)
e535c052 Perhaps gnulib for portability problems? Temp. disabled few tools.
c49af70c Few sample tests for tools in automated tests
02a08edf Include licensing blurb in every single file since it's very important
721ee498 Update docs a bit
56b426af First steps towards unbundling backend libraries
caa77047 Remove debug output inadvertently left enabled
6296b078 Fix sign-problem causing format-guessing for OFST transducers to fail
9444698e Add support for converting to the HFST optimized-lookup format
39bc89ea Add support for the HFST-optimized-lookup implementation in the tool for format conversion
134c6ea2 Add support for the new implementation in the interface files
0fd24ab1 Modify the semantics of the InputStream function is_good in the OFST implementations to match those in the SFST and foma implementations
4883d03a Update makefile for building the new backend
ec67b18c Add conversion HFST-optimized-lookup -> OFST Tropical Weight
41d7301d Initial commit of bridge classes for the HFST-optimized-lookup backend
f8b849b6 Initial commit of an HFST-optimized-lookup backend
3532dca8 Add is_cyclic to the transducer API and implementations
532c6683 Add hfst-strip-header tool for removing any HFST3 headers from a file
c7a64f3e Port hfst-compare to the new library
96b6ca71 Pass the correct output type when creating the output stream
29dc4894 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk471 941e2c2b-deac-454f-805a-451daa25f33c
5834ac0d Port hfst-fst2txt
9d0ade84 Fix the already-ported tools to properly handle IO using the standard streams
5cc51002 Add ability to read the type of the first transducer in an HfstInputStream without first having created a transducer from it
ab1d96cc git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk465 941e2c2b-deac-454f-805a-451daa25f33c
714fc13a HfstCompiler class added
7580d1d7 Fix fst2fst to correctly handle inputs with multiple transducers
e34e30c2 Teach HfstInputStream how to deal with inputs containing multiple transducers, of which none have HFST3 headers
50d56868 Port hfst-invert to HFST3
1a86ae31 Port hfst-fst2fst to work with HFST3
de1feba4 Add support in HfstInputStream for loading transducers lacking the HFST3 header
a5e7ec01 Make foma transducer type guesser able to detect non-gzipped transducers as well
7f785cff Add call to open the output stream so the tool doesn't crash
dc85825e Add static methods to the InputStream implementations allowing to check whether a FILE* or istream contains a transducer of a give type
1e6b6cd4 Remove HFST2 include from file with helper functions for tools
0ab49edd hfst-calculate files updated to hfst version 3. At the moment, hfst-calculate only parses single transition expressions and produces a dummy transducer as a result.
fbe8acb2 anonymous transducer constructors added
ef4f1c58 Added HfstTransducer &operator=(const HfstTransducer &)
e352b9ab hfst-compiler.yy added
544c4183 more rule operator functions implemented
6886485c git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk409 941e2c2b-deac-454f-805a-451daa25f33c
13ec6820 'HfstRules.cc' added.
d72acf32 rule functions added
cfe73d09 stub for flag diacritic support doc
916c12d7 documentation added
dbd6d6ed more Doxygen documentation added
9759312e Convert binary integers to decimal for old gcc support (OS X)
0067fa6f Tried to make the autoreconf text a bit clearer.
7ded1ccc ImplementationType parameters removed from most functions
91ac6d0c Require fst/fstlib.h as <fst/fstlib.h> and include as such.
55b3eb58 * Remove openfst from source directory because bundling Apache licenced software with GNU GPL softare is a bad thing * Add licence disclaimers to auxiliary files because FSF suggests so
c695e9af documentation added
49053036 Documentation and type handling added, some problems in type conversion.
7d5dbb0a add ofst include dir for compilation
f76eb453 sgi.h is visible outside libhfst
7bf66dd7 Work around problems with missing hash
c618bd71 Add extra doc stuff to build tree
df2f4126 more doxygen documentation added
8804a385 some typedefs in SymbolDefs.h and ExtractStrings.h renamed
a05ba5d3 copy hfst-proc verbatim to 3
ce06b138 Swap stable old trunk to branches and proposition to trunk


=================================
NEWS for hfst-tools and library
=================================

This file contains all noteworthy changes in HFST development between releases.
For full listing of changes see ChangeLog.

b942a23f Remove libtool 1 support tricks, more trouble than it's worth
86718c20 Mark version requirements in autogen
c5094f14 Move version requirement to beginning of the line to be sure...
95bd5252 debugging tool
063de32f A bug in function random_path fixed
77c7bedb help messages to message_out instead of stderr
a9faa385 Do not cut unknown word forms from first non-alphabetic character in analysis, test case added
700c106f Fixed a piece of documentation on HfstTransitionGraph and added a test case too.
e9f9a79b Now extract_random_paths works for an empty tropical transducer. A test case also added.
89e4743c print more than fstinfo only if verbose
0b38d0b0 Fix arc-based leak in conversion
68c0d397 Make old lexc default lexc again; fixes: 100000000 Mac OS X bugs
637ea552 allow options to be passed via env.var. HFST_OPTIONS
3cf23c83 works but leaks
77a1c0da and default to glib to test if end users are ready for that in next rel
20c88c93 Release changes
6ccfd1c1 delete created test files
ec441723 include new proc tests in dist
25d65308 extra dist m4 macros
ce67ad2c fix how compound words are generated
6c780612 try generating first without compound interpretations; possible fix for bug 3290585
b71cb2e6 do not pipe error messages from tests to /dev/null
a8b07206 fix some -a tests
267a212e add valgrind to stress test targets
4af5dca4 Add a blurb about Unicode support requirements
b2f03ac6 calculate spaces in length of print forms; fixes 3296854
0bd58aa3 Allow use of glib for proc unicode casing (most likely breaks all Macs because of pkg-config and all that)
06620843 Implement type mismatch error messages for all binary tools
dec54f25 add quotation and neat printing in twolc tests
6048824f incompatible format messages in composition
f43dfaf8 document new tests in troubleshooting with examples.
dc5c1c50 Move latin-1 string tests to XFAIL
c30f3d37 Neat error output when incompatible formats; bug 3287458
eb6e0ed9 Troubleshooting for Mac OS X's getopt
0325c9f1 Check for GNU compatible getopt, try to fallback to broken getopts more aggressively on Mac OS X
23223aff Implement print-sigma and acceptor
4a279dde include new command line option in man pages
8e4177d5 Disable reserved characters and escapes in raw mode.
3b5d30f2 remove roundtrip test: mac os x sed does not support new lines
6ce1996e hfst-lexc around lexc_read
df3b87ee more alphabet hacks, i should really fix this properly as opposed to just patching the bugs i find
83fe1009 New analysis mode for finntreebank people: * no lowercase * no uppercases * no compounds * probably some ohter things
c46b8e80 Detail all external backend libraries in troubleshooting section also...
332fb8dc here here
0a94bbc7 possibly hippu-compatible alias installation
917ae736 Now HfstTransducer::substitute(StringPair, HfstTransducer) does not modify the argument transducer. A test for this case also added.
795926d1 Provide preliminary autoconf scripts for HFST
52f08847 Replaced HfstFastTransducer with HfstBasicTransducer in conversions. This might make HFST slower, but it is needed until the bugs in alphabet handling are fixed.
2fd9cfbe When performing transducer conversion through HfstFastTransducer, symbols leaked from one alphabet to another. Now HfstBasicTransducer is used instead and a test case is added that will fail if HfstFastTransducer is used instead. The original bug is to be fixed at some point.
9779c9b5 Added a test case for replace_up with FOMA_TYPE that fails. Currently it is commented so that make check will pass, but it will be fixed soon.
12dfacfa Now binary operations HfstTransducer::disjunct etc. do not modify the argument transducer.
f933bbb9 HfstTransducer::insert_to_alphabet and HfstTransducer::remove_from_alphabet now pass the tests.
25fd77f5 Added function HfstTransducer::remove_from_alphabet. Currently throws FunctionNotImplementedException because it does not pass all tests.
5447d492 verbose test failures
1085c4f8 Documentation updated.
c6d0e148 Removed check for null-termination of string in the test of string_mainpultion.cc, because it fails on Mac.
49f1dff1 Now distcheck passes the tests but still complains about att files..
e8842434 Reverted r1418 - these files are supposed to be here, and 'make clean' is over-cleaning.
e7725b41 Deleted supposedly generated files - they were removed by the make clean target in test/.
62254319 assert fopen != NULL
cf16a6f6 Modified compose-intersect, so that it will treat only flg diacritics not found in the alphabet of the rules as epsilons.
80c2c7ab A small bug fixed in the newest version of foma in the foma svn. Now the FomaTransducer test works correctly.
ea2f6198 Code reorganized. There seems to be some errors in openfst's minimization, temporarily fixed the situation by pushing weights before minimization.
5aca45b4 Retain root joiners while applying morphotax
afa9e112 kill some warnings by settings
16eabd97 New NEWS
ae3d1932 New releasenum (3.1.1) and libvector (6:0:0)
aa7bf174 Changed the test script so that it removes temp.twolc.hfst0, when it's done
97268c20 Fixed typo where $SRCDIR wasn't getting assigned to the file we want to test
077da455 Changed [[ ... ]] in regular expression center rules to <[ ... ]> to avoid introducing syntax errors in previously working twolc-grammars.
7d9a60d1 Added new tests in Makefile.am.
97963cf5 Added support for rules which have centers that are regular expressions of pairs.
9ebc8b9f Added tests for rules with regular expression centers.
b583a0cc Improved test script so it won't exit normally if hfst-twolc crashes.
e358dbec The public and protected interface of HfstTransitionGraph revised. Unnecessary comments and debugging code removed from HfstTransducer.
e1ef196f class HfstTransition moved to file 'HfstTransition.h'
147b99d8 Substitute functions in HfstTransitionGraph should now be easier to read.
add8e685 Code of HfstTransitionGraph made clearer, a bug in including HfstTransitionData.h fixed.
dc9da7bb Code simplified.
4ad9718b Transducer conversion done through HfstTransitionGraph.
f23ba679 Included <algorithm>...
3856fdc3 Changed member const HfstBasicTransducer &t to HfstBasicTransducer t in ComposeIntersectFst
96e5e02a Fixed bug in ComposeIntersectFst which forced all states in a transducer to have the same final weight as the initial state.
6b74eb4b Sorting the arcs of the argument transducer in the constructor of ComposeIntersectFst, because they need not be sorted nowadays.
0a5bcef5 Added arc_sort member function to HfstTransitionGraph.
2f9d3a99 Some additional experimental container-hacking layer stuff
e238723a left arrow rules
f49a6e12 some more hacks to alphabet.cc
6a3b6422 Added support for rule centers which have the form 'X:a' where 'X' is a set and 'a' is a symbol.
972561a7 Added tests for rules with centers 'X:a' where 'X' is a set and 'a' is a symbol.
475b6f90 Undid some changes that stopped working after the interface of HfstBasicTransducer apparently changed. Also changed HfstTransducer OtherSymbolTransducer::get_transducer(void) to const and added the function void OtherSymbolTransducer::get_initial_transition_pairs(SymbolPairVector &pair_container) const.
48487f20 Made max() and is_subset() static functions.
83cfb8a8 hfst-twolc unit tests added.
3bdda4e9 New NEWS
d04a96b8 New release number, 3.1.0 release and 5:0:0 libvector
420e273d Removed an extra debugging print
c23e9199 HfstTransducer::insert_freely(HfstTransducer&) now uses HfstFastTransducer
1bc69629 check locale in some tools
f0633660 HfstTransducer::insert_freely now uses HfstFastTransducer for foma type. More tests also added for insert_freely.
54900958 Conversion functions between HfstFastTransducer and backend implementations added. Now HfstTransducer::convert also converts through HfstFastTransducer. HfstFastTransducer will gradually replace HfstBasicTransducer for HfstTransducer functions that require conversion.
79722bf1 HfstTransitionGraph now has a vector of states instead of a set. All programs and functions updated accordingly.
3b908de2 Reword slow lookup warning
ccad335b An apparently functional way to expose sets by copying them in order to vectors
8708944a Fixed typo
92ef2989 Fixed includes in XreCompiler.h and LexcCompiler.h
8fa24816 added some functions to ConversionFunctions that will be used in fast conversion between transducer types instead of HfstConstantTransducer
0aea7c99 Fixed test script so that it should work in distcheck
6ee3d240 Made lookdown functions consistent with lookup ones, updated docs
1c9307d5 Updated doxygen to reflect changes in lookup functions
5f283556 simple archive tool for duplicating repeats
a43ca610 Added a faster version of lexc compiler for a transducer of foma type.
228feee6 decrease stress by order of 8
b15d21d4 Keep information about the parent FdTable of an FdState in the class to avoid invalid reads if it gets deleted
153862dc Changed lookup functions to return pointer to HfstOneLevelPaths, and take no argument for it. Calling functions are (still) responsible for freeing memory.
577e62cf Add more common problems
cd64166d Added HfstTransducer::get_symbol_number, it will be used in faster conversion between transducer formats.
5588a3b4 A small bug in testing the EmptyStringException fixed.
bda6a257 EmptyStringException is now thrown if the empty string is used as a transition symbol.
6923dc2c Universal pair function added
92ece0da Need more inclusions and usings when they don't leak in from backend libraries
ed3be664 Pathnames of testfiles in test/libhfst are now given as getenv("pwd") + "/tesfilename"
a5e5bceb Tests in test/libhfst now create the test files in EXTRA_DIST, so that distcheck works. This is a temporary solution.
7c1efd37 Remove obvious memory leaks
e8bb1685 Added missing test files.

---------------------------

* Fixed inclusion path in pkg-config file

* Fixed extraneous epsilon path in hfst-lexc transducers

---------------------------------

* Multiple compilation issued fixed w.r.t string.h inclusion inside hfst source
tree without installed headers

* Added preliminary pkg-config support

* Optimizations for flag diacritics in optimized format; NB. the optimized
format in 2.4 is not compatible with optimized format of previous releases

* More input and output formatting options for hfst-lookup

* New tool hfst-omor-evaluate for testing morphological transducers
(an overblown glorified hfst-diff-test)

* Preliminary tool for killing evil flag diacritics

* Several minor bug fixes in composing intersection and flag diacritics