[hunspell] 08/98: Imported Upstream version 1.0.9
This is an automated email from the git hooks/post-receive script.
rene pushed a commit to branch master
in repository hunspell.
commit 9e90728d5d14f61d3094e0e6cb130cb698ee77ef
Author: Rene Engelhard <rene@debian.org>
Date: Thu Apr 21 14:45:09 2016 +0200
Imported Upstream version 1.0.9
---
AUTHORS.myspell | 48 +--
ChangeLog | 143 +++++++-
ChangeLog.O | 2 +
NEWS | 273 ++++++++++++++
THANKS | 1 +
TODO | 5 +-
configure | 23 +-
configure.ac | 7 +-
man/hu/hunspell.4 | 16 +
man/hunspell.4 | 14 +-
src/hunspell/Makefile.am | 2 +-
src/hunspell/Makefile.in | 2 +-
src/hunspell/affentry.cxx | 52 +--
src/hunspell/affentry.hxx | 5 +-
src/hunspell/affixmgr.cxx | 505 +++++++++++++++++---------
src/hunspell/affixmgr.hxx | 25 +-
src/hunspell/atypes.hxx | 3 +-
src/hunspell/csutil.cxx | 47 ++-
src/hunspell/csutil.hxx | 10 +-
src/hunspell/hashmgr.cxx | 27 +-
src/hunspell/hashmgr.hxx | 2 +
src/hunspell/hunspell.cxx | 187 ++++++----
src/hunspell/hunspell.hxx | 4 +-
src/hunspell/langnum.hxx | 6 +-
src/hunspell/makefile.mk | 112 ++++++
src/hunspell/suggestmgr.cxx | 750 +++++++++++++++++++++++++--------------
src/hunspell/suggestmgr.hxx | 65 ++--
src/tools/Makefile.am | 8 +-
src/tools/Makefile.in | 32 +-
src/tools/example.cxx | 89 +++++
src/tools/{unmunch.c => munch.c} | 430 +++++++++++++++++++---
src/tools/munch.h | 121 +++++++
src/tools/unmunch.c | 10 +-
tests/Makefile.am | 15 +-
tests/Makefile.in | 15 +-
tests/complexprefixes.aff | 9 +
tests/complexprefixes.dic | 3 +
tests/complexprefixes.good | 3 +
tests/complexprefixes.test | 4 +
tests/complexprefixes.wrong | 2 +
tests/conditionalprefix.morph | 14 +
tests/germancompounding.aff | 15 +-
tests/germancompounding.dic | 2 +-
tests/germancompounding.good | 8 +-
tests/germancompounding.wrong | 34 +-
tests/pseudoroot4.aff | 2 +
tests/pseudoroot4.dic | 5 +
tests/pseudoroot4.good | 5 +
tests/pseudoroot4.test | 4 +
49 files changed, 2403 insertions(+), 763 deletions(-)
diff --git a/AUTHORS.myspell b/AUTHORS.myspell
index f591c4d..7c7f980 100644
--- a/AUTHORS.myspell
+++ b/AUTHORS.myspell
@@ -18,7 +18,7 @@ to help improve and create suggestions for very poorly
spelled words.
Please send any and all contributions or improvements
-to him(me) or to dev@lingucomponent.openoffice.org.
+to him or to dev@lingucomponent.openoffice.org.
David Einstein (Deinst@world.std.com) developed an almost
@@ -30,13 +30,16 @@ compatibility and work on merging our versions of MySpell
back into a single tree. David has been a significant help
in improving MySpell.
-La'szlo' Ne'meth <nemethl@gyorsposta.hu> is the author of
-the Hungarian dictionary and he developed and contributed
-the code to support compound words in MySpell and fixed
-numerous problems with the encoding case conversion
-tables. He also introduced the idea and is the author of
-the code to support replacement tables which greatly
-improved suggestions.
+
+N�meth L�szl� <nemethl@gyorsposta.hu> is the author of
+the Hungarian dictionary and he developed and contributed
+extensive changes to MySpell including ...
+ * code to support compound words in MySpell
+ * fixed numerous problems with encoding case conversion tables.
+ * designed/developed replacement tables to improve suggestions
+ * changed affix file parsing to trees to greatly speed loading
+ * removed the need for malloc/free pairs in suffix_check which
+ speeds up spell checking in suffix rich languages by 20%
Davide Prina <davideprina@uahoo.com>, Giuseppe Modugno
<gppe.modugno@libero.it>, Gianluca Turconi <luctur@comeg.it>
@@ -54,8 +57,6 @@ that have greatly improved MySpell auggestions
by NOSPLITSUGS line in affix file)
-
-
Special Thanks to all others who have either contributed ideas or
testing for MySpell
@@ -64,30 +65,3 @@ Thanks,
Kevin Hendricks
kevin.hendricks@sympatico.ca
-
-
-
-
-
-
-
-
-
-
-Date: Thu Dec 4 17:24:53 2003
-To: "Kevin B. Hendricks" <kevin.hendricks@sympatico.ca>
-
-Hi Kevin,
-
-At 09:42 4-12-2003 -0500, you wrote:
->Hi Simon,
->
->The best way to do a full diff of two different directories (recursively) is:
->
->diff -urN myspell.orig/ myspell/ > simon_changes.txt
->
->And then gzip or zip simon_changes.txt and send it to me as an attachment.
-
-Here is the patch.
-
-It should include the following changes:
diff --git a/ChangeLog b/ChangeLog
index 5a26703..5278180 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,144 @@
+2005-08-26 Németh László <nemethl@gyorsposta.hu>:
+improvements:
+
+ * src/hunspell/suggestmgr.cxx:
+ Unicode support in related character map suggestion
+
+ * src/hunspell/suggestmgr.cxx: Unicode support in ngram suggestion
+
+ * src/hunspell/{suggestmgr,affixmgr,hunspell}.cxx: improve ngram suggestion.
+ Fix http://qa.openoffice.org/issues/show_bug.cgi?id=35725. See release
+ notes for examples. This problem reported by beccablain at OpenOffice.org.
+ - ngram suggestions now are case insensitive (see `Permenant' bug in Issuezilla)
+ - weight ngram suggestions (with the longest common subsequent algorithm,
+ also considering lengths of bad word and suggestion, identical first
+ letters and almost completely identical character positions)
+ - set strict affix congruency in expand_rootword(). Now ngram suggestions
+ are good for languages with rich morphology and also better for English.
+ Rationale: affixed forms of the first ngram suggestion
+ very often suppress the second and subsequent root word suggestions. But
+ faults in affixes are more uncommon, and can be fix without suggestions.
+ We must prefer the more informative second and subsequent root word
+ suggestions instead of the suggestions for bad affixes.
+ - a better suggestion may not be substring of a less good suggestion
+ Rationale: Suggesting affixed forms of a root word is
+ unnecessary, when root word has got better weighted ngram value.
+ (Checking substrings is a good approximation for this refinement.)
+ - lesser ngram suggestions (default 3 maximum instead of 10)
+ Rationale: For users need a big extra effort to check a lot of bad ngram
+ suggestions, nine times out of ten unnecessarily. It is very
+ distracting, because ngram suggestions could be very different.
+ Usually Myspell and Hunspell suggest one or two suggestions with
+ the old suggestion algorithms (maximum is 15), with ngram algorithm
+ often gives maximum number suggestions. With strict affix congruency
+ and other refinements, the good suggestion there is usually among the
+ first three elements.
+ - new affix parameter: MAXNGRAMSUG
+
+ * src/hunspell/*: support agglutinative languages with rich prefix
+ morphology or with right-to-left writing system (for example, Turkic
+ and Austronesian languages with (modified) Arabic scripts).
+ - new affix parameter: COMPLEXPREFIXES
+ Set twofold prefix stripping (but single suffix stripping)
+ * src/hunspell/affixmgr.cxx:
+ - speed up prefix loading with tree sorting algorithm.
+ * tests/complexprefixes.*, tests/complexprefixesutf.*:
+ Coptic example posted by Moheb Mekhaiel
+
+ * src/hunspell/hashmgr.cxx: check size attribute in dic file
+ suggested by Daniel Naber
+ Rationale: With missing size attribute Hunspell allocates too small and
+ more slower hash memory, and Hunspell can lose first dictionary word.
+
+ * src/hunspell/affixmgr.cxx: check stripping characters and condition
+ compatibility in affix rules (bugs detected in cs_CZ, es_ES, es_NEW,
+ es_MX, lt_LT, nn_NO, pt_PT, ro_RO and sk_SK dictionaries). See release
+ notes of Hunspell 1.0.9 in NEWS.
+
+ * src/hunspell/affixmgr.cxx: check unnecessary fields in affix rules
+ (bugs detected in ro_RO and sv_SE dictionaries). See release notes.
+
+ * src/hunspell/affixmgr.cxx: remove redundant condition checking
+ in affix rules with stripping characters (redundancy in OpenOffice.org
+ dictionaries reported by Eleonóra Goldman)
+ Rationale: this is a little optimization, but it was excellent for
+ detect the bad ngram affixation with bad or weak affix conditions.
+
+ * tests/germancompounding.aff: improve compound definition
+ - use dash prefix instead of language specific tokenizer
+ Rationale: Using uniform approach is the right way to check and analyze
+ compound words. Language specific word breaking is deprecated, need
+ a sophisticated grammar checking for word-like word pairs
+ (for example in Hungarian there is a substandard, but accepted
+ syntax with dash for word pairs: cats, dogs -> kutyák-macskák (like
+ cats/dogs in English).
+
+ * test Hunspell with 54 OpenOffice.org dictionaries: see release notes
+
+bug fixes:
+
+ * src/hunspell/suggestmgr.*: add time limit to exponential
+ algorithm of the related character map suggestion
+ Rationale: a long word in agglutinative languages or a special pattern
+ (for example a horizontal rule) made of map characters can `crash' the
+ spell checker.
+
+ * src/hunspell/affentry.cxx: add() functions: fix bad word generation
+ checking stripping characters (see similar bug in unmunch)
+
+ * src/hunspell/affixmgr.cxx: parse_file(): fix unconditional getNext()
+ call for ~AffixMgr() when affix file is corrupt.
+
+ * src/hunspell/affixmgr.*: AffixMgr(), parse_cpdsyllable(): fix missing
+ string duplications for ~AffixMgr() when affix file is corrupt.
+
+ * src/hunspell/affixmgr.*: parse_affix(): fix fprintf() call when affix
+ file is corrupt. Bug reported by Daniel Naber.
+
+ * suggestmgr.cxx: replace single usage of 'strdup' with 'mystrdup'
+ patch by Chris Halls (debian.org)
+
+ * src/hunspell/makefile.mk: add makefile.mk for compiling in OpenOffice.org
+ See README in Hunspell UNO modul.
+ Problems with separated compiling reported by Rene Engelhard
+
+ * src/hunspell/hunspell.cxx: fix pseudoroot support
+ - search a not pseudoroot homonym in check()
+ * tests/pseudoroot4.*: test this fix
+
+ * src/tools/unmunch.c: fix bad word generation when conditions
+ are shorter or incompatible with stripping characters in affix rules
+
+ * src/tools/unmunch.c: fix mychomp() for de_AT.dic and other dic files
+ without last new line character.
+
+other changes:
+ * src/hunspell/suggestmgr.*: erase ACCENT suggestion
+ Rationale: ACCENT suggestion was the same as Kevin Hendrick's map
+ suggestion algorithm, but with a less good interface in affix file.
+
+ * src/hunspell/suggestmgr.*: combine cycle number limit
+ in badchar(), and forgotchar() with a time limit.
+
+ * src/hunspell/affixmgr.*: remove NOMAPSUGS affix parameter
+
+ * src/hunspell/{suggestmgr,hunspell}.*: strip periods from
+ suggestions (restore MySpell's original behaviour)
+ Rationale: OpenOffice.org has an automatic period handling mechanism
+ and suggestions look better without periods.
+ - new affix file parameter: SUGSWITHDOTS
+ Add period(s) to suggestions, if input word terminates in period(s).
+ (No need for OpenOffice.org dictionaries.)
+
+ * tests/germancompounding.aff: improve bad german affix in affix example
+ (computeren->computern). Suggested by Daniel Naber.
+
+ * src/tools/example.cxx: add Myspell's example
+
+ * src/tools/munch.cxx: add Myspell's munch
+
+ * man{,/hu}/hunspell.4: refresh manual pages
+
2005-08-01 Németh László <nemethl@gyorsposta.hu>:
* add missing MySpell files and features:
- add MySpell license.readme, README and CONTRIBUTORS ({license,README,AUTHORS}.myspell)
@@ -10,7 +151,7 @@
* src/man/man.4: MAP, COMPOUNDPERMITFLAG, NOSPLITSUGS, NOMAPSUGS
* src/hunspell/aff{entry,ixmgr}.cxx:
- - improved compound word support
+ - improve compound word support
- new affix parameter: COMPOUNDPERMITFLAG (see manual)
* src/tests/compoundaffix{,2}.*: examples for COMPOUNDPERMITFLAG
* src/tests/germancompounding.*: new solution for German compounding
diff --git a/ChangeLog.O b/ChangeLog.O
index d9e911a..a2c712d 100644
--- a/ChangeLog.O
+++ b/ChangeLog.O
@@ -1,3 +1,5 @@
+Myspell has a lot of parallel development, that is not documented here.
+
2005-01-11: N�meth L�szl� <nemethl@gyorsposta.hu>
* hunspell.cxx:
- interakt�v jav�t�sn�l hi�nyz� �j sor karakterek p�tl�sa.
diff --git a/NEWS b/NEWS
index 50082c9..6d6c7f5 100644
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,276 @@
+2005-08-26: Hunspell 1.0.9 release
+
+* improved related character map suggestion
+
+* improved ngram suggestion
+
+------ examples for ngram improvement (O=old, N = new ngram suggestions) --
+
+1. Permenant (instead of Permanent)
+
+O: Endangerment, Ferment, Fermented, Deferment's, Empowerment,
+ Ferment's, Ferments, Fermenting, Countermen, Weathermen
+
+N: Permanent, Supermen, Preferment
+
+Note: Ngram suggestions was case sensitive.
+
+2. permenant (instead of permanent)
+
+O: supermen, newspapermen, empowerment, endangerment, preferments,
+ preferment, permanent, preferment's, permanently, impermanent
+
+N: permanent, supermen, preferment
+
+Note: new suggestions are also weighted with longest common subsequence,
+first letter and common character positions
+
+3. pernemant (instead of permanent)
+
+O: pimpernel's, pimpernel, pimpernels, permanently, permanents, permanent,
+ supernatant, impermanent, semipermanent, impermanently
+
+N: permanent, supernatant, pimpernel
+
+Note: new method also prefers root word instead of not
+relevant affixes ('s, s and ly)
+
+
+4. pernament (instead of permanent)
+
+O: tournaments, tournament, ornaments, ornament's, ornamenting, ornamented,
+ ornament, ornamentals, ornamental, ornamentally
+
+N: ornamental, ornament, tournament
+
+Note: Both ngram methods misses here.
+
+
+5. obvus (instad of obvious):
+
+O: obvious, Corvus, obverse, obviously, Jacobus, obtuser, obtuse,
+ obviates, obviate, Travus
+
+N: obvious, obtuse, obverse
+
+Note: new method also prefers common first letters.
+
+
+6. unambigus (instead of unambiguous)
+
+O: unambiguous, unambiguity, unambiguously, ambiguously, ambiguous,
+ unambitious, ambiguities, ambiguousness
+
+N: unambiguous, unambiguity, unambitious
+
+
+
+7. consecvence (instead of consequence)
+
+O: consecutive, consecutively, consecutiveness, nonconsecutive, consequence,
+ consecutiveness's, convenience's, consistences, consistence
+
+N: consequence, consecutive, consecrates
+
+
+An example in a language with rich morphology:
+
+8. Misisipiben (instead of Mississippiben [`in Mississippi' in Hungarian]):
+
+O: Misik�d�iben, Pisised�iben, Misik�i�iben, Pisisek�iben, Misik�iben,
+ Misik�id�iben, Misik�k�iben, Misik�ik�iben, Misik�im�iben, Mississippiiben
+
+N: Mississippiben, Mississippiiben, Misiiben
+
+Note: Suggesting not relevant affixes was the biggest fault in ngram
+ suggestion for languages with a lot of affixes.
+
+--------------- end of examples --------------------
+
+* support twofold prefix cutting
+
+* lots of other improvements and bug fixes (see ChangeLog)
+
+* test Hunspell with 54 OpenOffice.org dictionaries:
+
+source: ftp://ftp.services.openoffice.org/pub/OpenOffice.org/contrib/dictionaries
+
+testing shell script:
+-------------------------------------------------------
+for i in `ls *zip | grep '^[a-z]*_[A-Z]*[.]'`
+do
+ dic=`basename $i .zip`
+ mkdir $dic
+ echo unzip $dic
+ unzip -d $dic $i 2>/dev/null
+ cd $dic
+ echo unmunch and test $dic
+ unmunch $dic.dic $dic.aff 2>/dev/null | awk '{print$0"\t"}' |
+ hunspell -d $dic -l -1 >$dic.result 2>$dic.err || rm -f $dic.result
+ cd ..
+done
+--------------------------------------------------------
+
+test result (0 size is o.k.):
+
+$ for i in *_*/*.result; do wc -c $i; done
+0 af_ZA/af_ZA.result
+0 bg_BG/bg_BG.result
+0 ca_ES/ca_ES.result
+0 cy_GB/cy_GB.result
+0 cs_CZ/cs_CZ.result
+0 da_DK/da_DK.result
+0 de_AT/de_AT.result
+0 de_CH/de_CH.result
+0 de_DE/de_DE.result
+0 el_GR/el_GR.result
+6 en_AU/en_AU.result
+0 en_CA/en_CA.result
+0 en_GB/en_GB.result
+0 en_NZ/en_NZ.result
+0 en_US/en_US.result
+0 eo_EO/eo_EO.result
+0 es_ES/es_ES.result
+0 es_MX/es_MX.result
+0 es_NEW/es_NEW.result
+0 fo_FO/fo_FO.result
+0 fr_FR/fr_FR.result
+0 ga_IE/ga_IE.result
+0 gd_GB/gd_GB.result
+0 gl_ES/gl_ES.result
+0 he_IL/he_IL.result
+0 hr_HR/hr_HR.result
+200694989 hu_HU/hu_HU.result
+0 id_ID/id_ID.result
+0 it_IT/it_IT.result
+0 ku_TR/ku_TR.result
+0 lt_LT/lt_LT.result
+0 lv_LV/lv_LV.result
+0 mg_MG/mg_MG.result
+0 mi_NZ/mi_NZ.result
+0 ms_MY/ms_MY.result
+0 nb_NO/nb_NO.result
+0 nl_NL/nl_NL.result
+0 nn_NO/nn_NO.result
+0 ny_MW/ny_MW.result
+0 pl_PL/pl_PL.result
+0 pt_BR/pt_BR.result
+0 pt_PT/pt_PT.result
+0 ro_RO/ro_RO.result
+0 ru_RU/ru_RU.result
+0 rw_RW/rw_RW.result
+0 sk_SK/sk_SK.result
+0 sl_SI/sl_SI.result
+0 sv_SE/sv_SE.result
+0 sw_KE/sw_KE.result
+0 tet_ID/tet_ID.result
+0 tl_PH/tl_PH.result
+0 tn_ZA/tn_ZA.result
+0 uk_UA/uk_UA.result
+0 zu_ZA/zu_ZA.result
+
+In en_AU dictionary, there is an abbrevation with two dots (`eqn..'), but
+`eqn.' is missing. Presumably it is a dictionary bug. Myspell also
+haven't accepted it.
+
+Hungarian dictionary contains pseudoroots and forbidden words.
+Unmunch haven't supported these features yet, and generates bad words, too.
+
+* check affix rules and OOo dictionaries. Detected bugs in cs_CZ,
+es_ES, es_NEW, es_MX, lt_LT, nn_NO, pt_PT, ro_RO, sk_SK and sv_SE dictionaries).
+
+Details:
+--------------------------------------------------------
+cs_CZ
+warning - incompatible stripping characters and condition:
+SFX D us ech [^ighk]os
+SFX D us y [^i]os
+SFX Q os ech [^ghk]es
+SFX M o ech [^ghkei]a
+SFX J �m ej �m
+SFX J �m ejme �m
+SFX J �m ejte �m
+SFX A ou�it up oupit
+SFX A ou�it upme oupit
+SFX A ou�it upte oupit
+SFX A nout l [aeiouy��������r][^aeiouy��������rl][^aeiouy
+SFX A nout l [aeiouy��������r][^aeiouy��������rl][^aeiouy
+
+es_ES
+warning - incompatible stripping characters and condition:
+SFX W umar �se [ae]husar
+SFX W emir i��is e�ir
+
+es_NEW
+warning - incompatible stripping characters and condition:
+SFX I unan �nen unar
+
+es_MX
+warning - incompatible stripping characters and condition:
+SFX A a ote e
+SFX W umar �se [ae]husar
+SFX W emir i��is e�ir
+
+lt_LT
+warning - incompatible stripping characters and condition:
+SFX U ti siuosi tis
+SFX U ti siuosi tis
+SFX U ti siesi tis
+SFX U ti siesi tis
+SFX U ti sis tis
+SFX U ti sis tis
+SFX U ti sim�s tis
+SFX U ti sim�s tis
+SFX U ti sit�s tis
+SFX U ti sit�s tis
+
+nn_NO
+warning - incompatible stripping characters and condition:
+SFX D ar rar [^fmk]er
+SFX U �re orde ere
+SFX U �re ort ere
+
+pt_PT
+warning - incompatible stripping characters and condition:
+SFX g �os oas �o
+SFX g �os oas �o
+
+ro_RO
+warning - bad field number:
+SFX L 0 le [^cg] i
+SFX L 0 i [cg] i
+SFX U 0 i [^i] ii
+warning - incompatible stripping characters and condition:
+SFX P l i l [<- there is an unnecessary tabulator here)
+SFX I a ii [gc] a
+warning - bad field number:
+SFX I a ii [gc] a
+SFX I a ei [^cg] a
+
+sk_SK
+warning - incompatible stripping characters and condition:
+SFX T �a� ol� kla�
+SFX T �a� ol�c kla�
+SFX T s�a� �l� sla�
+SFX T s�a� �l�c sla�
+SFX R �c� l�iem �c�
+SFX R i�s� �tie mias�
+SFX R iez� iem [^i]ez�
+SFX R iez� ie� [^i]ez�
+SFX R iez� ie [^i]ez�
+SFX R iez� eme [^i]ez�
+SFX R iez� ete [^i]ez�
+SFX R iez� � [^i]ez�
+SFX R iez� �c [^i]ez�
+SFX R iez� z [^i]ez�
+SFX R iez� me [^i]ez�
+SFX R iez� te [^i]ez�
+
+sv_SE
+warning - bad field number:
+SFX C 0 net nets [^e]n
+--------------------------------------------------------
+
2005-08-01: Hunspell 1.0.8 release
- improved compound word support
diff --git a/THANKS b/THANKS
index b1f2d24..21f85ee 100644
--- a/THANKS
+++ b/THANKS
@@ -10,6 +10,7 @@ Kevin Hendricks
Khiraly
Koblinger Egmont
Kornai András
+Bram Moolenaar
Daniel Naber
Noll János
Sarlós Tamás
diff --git a/TODO b/TODO
index 0d1b280..3109034 100644
--- a/TODO
+++ b/TODO
@@ -1,7 +1,10 @@
+* shared dictionaries for multi-user environment
+* check less conditions
* new data structure for morphological analysis
* implement morphological generation
* improve compound handling
* implement complete stemming
-* need a better time limit in suggestion manager (eg. using time.h)
* Unicode tokenization (for Hunspell program)
* handle different encodings (for Hunspell program)
+* Unicode unmunch (munch)
+* forbiddenword and pseudoword support in unmunch
diff --git a/configure b/configure
index 10298a8..6fc3191 100755
--- a/configure
+++ b/configure
@@ -1,6 +1,6 @@
#! /bin/sh
# Guess values for system-dependent variables and create Makefiles.
-# Generated by GNU Autoconf 2.59 for hunspell 1.0.8.
+# Generated by GNU Autoconf 2.59 for hunspell 1.0.9.
#
# Report bugs to <nemeth@mokk.bme.hu>.
#
@@ -269,8 +269,8 @@ SHELL=${CONFIG_SHELL-/bin/sh}
# Identity of this package.
PACKAGE_NAME='hunspell'
PACKAGE_TARNAME='hunspell'
-PACKAGE_VERSION='1.0.8'
-PACKAGE_STRING='hunspell 1.0.8'
+PACKAGE_VERSION='1.0.9'
+PACKAGE_STRING='hunspell 1.0.9'
PACKAGE_BUGREPORT='nemeth@mokk.bme.hu'
ac_unique_file="config.h.in"
@@ -788,7 +788,7 @@ if test "$ac_init_help" = "long"; then
# Omit some internal or obsolete options to make the list less imposing.
# This message is too long to be a string in the A/UX 3.1 sh.
cat <<_ACEOF
-\`configure' configures hunspell 1.0.8 to adapt to many kinds of systems.
+\`configure' configures hunspell 1.0.9 to adapt to many kinds of systems.
Usage: $0 [OPTION]... [VAR=VALUE]...
@@ -854,7 +854,7 @@ fi
if test -n "$ac_init_help"; then
case $ac_init_help in
- short | recursive ) echo "Configuration of hunspell 1.0.8:";;
+ short | recursive ) echo "Configuration of hunspell 1.0.9:";;
esac
cat <<\_ACEOF
@@ -987,7 +987,7 @@ fi
test -n "$ac_init_help" && exit 0
if $ac_init_version; then
cat <<\_ACEOF
-hunspell configure 1.0.8
+hunspell configure 1.0.9
generated by GNU Autoconf 2.59
Copyright (C) 2003 Free Software Foundation, Inc.
@@ -1001,7 +1001,7 @@ cat >&5 <<_ACEOF
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
-It was created by hunspell $as_me 1.0.8, which was
+It was created by hunspell $as_me 1.0.9, which was
generated by GNU Autoconf 2.59. Invocation command line was
$ $0 $@
@@ -1644,7 +1644,7 @@ fi
# Define the identity of the package.
PACKAGE=hunspell
- VERSION=1.0.8
+ VERSION=1.0.9
cat >>confdefs.h <<_ACEOF
@@ -3433,9 +3433,6 @@ fi
# Checks for libraries.
# Checks for header files.
-# libintl.h src/hunspell/hunspell.cxx:63
-# locale.h src/hunspell/hunspell.cxx:62
-# TODO: do not check for this header files, if no hunspell needed
ac_ext=c
@@ -9055,7 +9052,7 @@ _ASBOX
} >&5
cat >&5 <<_CSEOF
-This file was extended by hunspell $as_me 1.0.8, which was
+This file was extended by hunspell $as_me 1.0.9, which was
generated by GNU Autoconf 2.59. Invocation command line was
CONFIG_FILES = $CONFIG_FILES
@@ -9118,7 +9115,7 @@ _ACEOF
cat >>$CONFIG_STATUS <<_ACEOF
ac_cs_version="\\
-hunspell config.status 1.0.8
+hunspell config.status 1.0.9
configured by $0, generated by GNU Autoconf 2.59,
with options \\"`echo "$ac_configure_args" | sed 's/[\\""\`\$]/\\\\&/g'`\\"
diff --git a/configure.ac b/configure.ac
index 7efb285..ce795fe 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4,8 +4,8 @@
m4_pattern_allow
AC_PREREQ(2.59)
-AC_INIT(hunspell, 1.0.8, nemeth@mokk.bme.hu)
-AM_INIT_AUTOMAKE(hunspell, 1.0.8)
+AC_INIT(hunspell, 1.0.9, nemeth@mokk.bme.hu)
+AM_INIT_AUTOMAKE(hunspell, 1.0.9)
AC_CONFIG_SRCDIR([config.h.in])
AC_CONFIG_HEADER([config.h])
@@ -19,9 +19,6 @@ AC_PROG_RANLIB
# Checks for libraries.
# Checks for header files.
-# libintl.h src/hunspell/hunspell.cxx:63
-# locale.h src/hunspell/hunspell.cxx:62
-# TODO: do not check for this header files, if no hunspell needed
AC_CHECK_HEADERS([fcntl.h libintl.h locale.h unistd.h error.h])
diff --git a/man/hu/hunspell.4 b/man/hu/hunspell.4
index d579d91..20aa0b7 100644
--- a/man/hu/hunspell.4
+++ b/man/hu/hunspell.4
@@ -104,10 +104,22 @@ ISO8859\-14, KOI8-R, KOI8-U, microsoft-cp1251, ISCII-DEVANAGARI.
.IP "TRY karakterek"
A javaslattev�sn�l az 1 karakteres pr�b�lkoz�sokn�l az itt felsorolt
karakterekre cser�l, illetve b�v�t.
+.IP "NOSPLITSUGS"
+Nem javasol sz�felbont�st a Hunspell ezzel a be�ll�t�ssal.
+.IP "MAXNGRAMSUGS"
+Az ngram javaslatok maxim�lis sz�m�nak be�ll�t�sa. Alap�rtelmezetten
+3. A 0 �rt�k kikapcsolja az ngram javaslatokat.
+.IP "SUGSWITHDOTS"
+Ha a hib�s sz� pont(ok)ra v�gz�dik, ezeket a javaslatok eset�ben is
+felt�nteti. Az OpenOffice.org sz�t�rak sz�m�ra nem sz�ks�ges ez a
+be�ll�t�s.
.IP "LANG nyelvk�d"
Nyelv megad�sa (Morphbase kieg�sz�t�s, ami a nyelvf�gg�
programr�szek bekapcsol�s�t v�gzi). �j nyelv hozz�ad�sa a langnum.hxx
�s a csutil.cxx �llom�nyon kereszt�l lehets�ges a forr�sk�dban.
+.IP "COMPLEXPREFIXES"
+K�tszeres prefixum lev�g�s be�ll�t�sa (de ezzel a k�tszeres szuffixum
+lev�g�s lehet�s�ge megsz�nik).
.IP "COMPOUNDMIN sz�m"
A legkisebb sz�hossz, ami m�g �sszetett sz�ban szerepelhet.
Alap�rtelmez�s szerint 3 karakter.
@@ -129,6 +141,10 @@ kapcsol
.IP "CIRCUMFIX kapcsol�"
Ezzel a kapcsol�val jel�lt szuffixum csak ugyanilyen kapcsol�val
jel�lt prefixummal jelenhet meg egy t�v�n. (Pl. leg-�des-ebb).
+.IP "COMPOUNDPERMITFLAG kapcsol�"
+Alap�etelmez�s szerint a prefixumok �s szuffixumok nem fordulhatnak
+el� az �sszetett szavak belsej�ben. A COMPOUNDPERMITFLAG
+kapcsol�val rendelkez� affixumok viszont itt is megjelenhetnek.
.IP "COMPOUNDFORBIDFLAG kapcsol�"
Toldal�kolt szavak sz��sszet�telben val� szerepl�s�t tilt�
kapcsol�. A tilt�shoz az adott SFX szab�ly folytat�si
diff --git a/man/hunspell.4 b/man/hunspell.4
index ec8b585..4c37fbb 100644
--- a/man/hunspell.4
+++ b/man/hunspell.4
@@ -90,9 +90,13 @@ ISO8859\-14, KOI8-R, KOI8-U, microsoft-cp1251, ISCII-DEVANAGARI.
Hunspell can suggest right word forms, when those differs from the
bad form by one TRY character. Parameter of TRY is case sensitive.
.IP "NOSPLITSUGS"
-Optionally disabling split-word suggestions.
-.IP "NOMAPSUGS"
-Optionally disabling map suggestions.
+Disable split-word suggestions.
+.IP "MAXNGRAMSUGS num"
+Set number of n-gram suggestions. Value 0 switches off the n-gram suggestions.
+.IP "SUGSWITHDOTS"
+Add dot(s) to suggestions, if input word terminates in dot(s).
+(Not for OpenOffice.org dictionaries, because OpenOffice.org
+has an automatic dot expansion mechanism.)
.IP "LANG langcode"
Set language code. In Hunspell may be language specific codes enabled
by LANG code. At present there are az_AZ, de_DE, hu_HU, TR_tr specific
@@ -102,6 +106,9 @@ Set flag type. Default value is character. The `long' value sets
the 2-character flag type, the `num' sets the decimal number flag type.
Decimal flags numbered from 1 to 65535, and in flag fields are
separated by comma.
+.IP "COMPLEXPREFIXES"
+Set twofold prefix stripping (but single suffix stripping) for agglutinative
+languages with right-to-left writing system.
.IP "COMPOUNDMIN num"
Minimum length of words in compound words.
Default value is 3 letter.
@@ -210,7 +217,6 @@ MAP u
.fi
.RE
.PP
-This doesn't work with UTF-8 encoding.
.IP "PFX flag cross_product number"
.IP "PFX flag stripping prefix condition morphological_description"
.IP "SFX flag cross_product number"
diff --git a/src/hunspell/Makefile.am b/src/hunspell/Makefile.am
index db8eb6a..27b2e2a 100644
--- a/src/hunspell/Makefile.am
+++ b/src/hunspell/Makefile.am
@@ -7,4 +7,4 @@ include_HEADERS=affentry.hxx htypes.hxx affixmgr.hxx \
csutil.hxx hunspell.hxx atypes.hxx dictmgr.hxx \
suggestmgr.hxx baseaffix.hxx hashmgr.hxx langnum.hxx
-EXTRA_DIST=hunspell.dsp
+EXTRA_DIST=hunspell.dsp makefile.mk
diff --git a/src/hunspell/Makefile.in b/src/hunspell/Makefile.in
index 45916c3..3087aeb 100644
--- a/src/hunspell/Makefile.in
+++ b/src/hunspell/Makefile.in
@@ -207,7 +207,7 @@ include_HEADERS = affentry.hxx htypes.hxx affixmgr.hxx \
csutil.hxx hunspell.hxx atypes.hxx dictmgr.hxx \
suggestmgr.hxx baseaffix.hxx hashmgr.hxx langnum.hxx
-EXTRA_DIST = hunspell.dsp
+EXTRA_DIST = hunspell.dsp makefile.mk
all: all-am
.SUFFIXES:
diff --git a/src/hunspell/affentry.cxx b/src/hunspell/affentry.cxx
index 7e7df98..2bdab90 100644
--- a/src/hunspell/affentry.cxx
+++ b/src/hunspell/affentry.cxx
@@ -50,7 +50,7 @@ PfxEntry::~PfxEntry()
strip = NULL;
if (utf8) {
for (int i = 0; i < 8; i++) {
- if (conds.utf8.wchars[i]) free(conds.utf8.wchars[i]);
+ if (conds.utf8.wchars[i]) free(conds.utf8.wchars[i]);
}
}
if (morphcode) free(morphcode);
@@ -63,30 +63,22 @@ char * PfxEntry::add(const char * word, int len)
int cond;
char tword[MAXWORDLEN+1];
- /* make sure all conditions match */
- if ((len > stripl) && (len >= numconds)) {
- unsigned char * cp = (unsigned char *) word;
- for (cond = 0; cond < numconds; cond++) {
- if ((conds.base[*cp++] & (1 << cond)) == 0) // XXX UTF-8???
- break;
- }
- if (cond >= numconds) {
- /* we have a match so add prefix */
- int tlen = 0;
+ if ((len > stripl) && (len >= numconds) && test_condition(word) &&
+ (!stripl || (strncmp(word, strip, stripl) == 0))) {
+ /* we have a match so add prefix */
+ char * pp = tword;
if (appndl) {
- strcpy(tword,appnd);
- tlen += appndl;
- }
- char * pp = tword + tlen;
+ strcpy(tword,appnd);
+ pp += appndl;
+ }
strcpy(pp, (word + stripl));
return mystrdup(tword);
- }
}
return NULL;
}
-inline int PfxEntry::test_condition(char * st)
+inline int PfxEntry::test_condition(const char * st)
{
int cond;
unsigned char * cp = (unsigned char *)st;
@@ -406,32 +398,22 @@ char * SfxEntry::add(const char * word, int len)
char tword[MAXWORDLEN+1];
/* make sure all conditions match */
- if ((len > stripl) && (len >= numconds)) {
- unsigned char * cp = (unsigned char *) (word + len);
- for (cond = numconds; --cond >=0; ) {
- if ((conds.base[*--cp] & (1 << cond)) == 0)
- break;
- }
- if (cond < 0) {
+ if ((len > stripl) && (len >= numconds) && test_condition(word + len, word) &&
+ (!stripl || (strcmp(word + len - stripl, strip) == 0))) {
/* we have a match so add suffix */
strcpy(tword,word);
- int tlen = len;
- if (stripl) {
- tlen -= stripl;
- }
- char * pp = (tword + tlen);
if (appndl) {
- strcpy(pp,appnd);
- tlen += appndl;
- } else *pp = '\0';
- return mystrdup(tword);
- }
+ strcpy(tword + len - stripl, appnd);
+ } else {
+ *(tword + len - stripl) = '\0';
+ }
+ return mystrdup(tword);
}
return NULL;
}
-inline int SfxEntry::test_condition(char * st, char * beg)
+inline int SfxEntry::test_condition(const char * st, const char * beg)
{
int cond;
unsigned char * cp = (unsigned char *) st;
diff --git a/src/hunspell/affentry.hxx b/src/hunspell/affentry.hxx
index 2dbd7a0..254819e 100644
--- a/src/hunspell/affentry.hxx
+++ b/src/hunspell/affentry.hxx
@@ -37,6 +37,7 @@ public:
inline const char * getKey() { return appnd; }
char * add(const char * word, int len);
+ inline short getKeyLen() { return appndl; }
inline const char * getMorph() { return morphcode; }
@@ -53,7 +54,7 @@ public:
inline void setNextEQ(PfxEntry * ptr) { nexteq = ptr; }
inline void setFlgNxt(PfxEntry * ptr) { flgnxt = ptr; }
- inline int test_condition(char * st);
+ inline int test_condition(const char * st);
};
@@ -121,7 +122,7 @@ public:
inline void setNextEQ(SfxEntry * ptr) { nexteq = ptr; }
inline void setFlgNxt(SfxEntry * ptr) { flgnxt = ptr; }
- inline int test_condition(char * st, char * begin);
+ inline int test_condition(const char * st, const char * begin);
};
#endif
diff --git a/src/hunspell/affixmgr.cxx b/src/hunspell/affixmgr.cxx
index 219cf8f..03e8a32 100644
--- a/src/hunspell/affixmgr.cxx
+++ b/src/hunspell/affixmgr.cxx
@@ -24,6 +24,8 @@ AffixMgr::AffixMgr(const char * affpath, HashMgr* ptr)
encoding=NULL;
utf8 = 0;
utf_tbl = NULL;
+ complexprefixes = 0;
+ maptable = NULL;
reptable = NULL;
numrep = 0;
compoundflag = FLAG_NULL; // permits word in compound forms
@@ -47,7 +49,6 @@ AffixMgr::AffixMgr(const char * affpath, HashMgr* ptr)
pfxappnd=NULL; // previous prefix for counting the syllables of prefix BUG
sfxappnd=NULL; // previous suffix for counting a special syllables BUG
cpdsyllablenum=NULL; // syllable count incrementing flag
- accent=NULL; // accented letters
checknum=0; // checking numbers, and word with numbers
wordchars=NULL; // letters + spec. word characters
version=NULL; // affix and dictionary file version string
@@ -58,8 +59,13 @@ AffixMgr::AffixMgr(const char * affpath, HashMgr* ptr)
circumfix = FLAG_NULL;
onlyincompound = FLAG_NULL;
flag_mode = FLAG_CHAR; // default one-character flags in affix and dic file
- nomapsugs = 0;
+ maxngramsugs = -1; // undefined
nosplitsugs = 0;
+ sugswithdots = 0;
+
+ derived = NULL; // XXX not threadsafe variable for experimental stemming
+ sfx = NULL;
+ pfx = NULL;
for (int i=0; i < SETSIZE; i++) {
pStart[i] = NULL;
@@ -75,13 +81,9 @@ AffixMgr::AffixMgr(const char * affpath, HashMgr* ptr)
if (parse_file(affpath)) {
fprintf(stderr,"Failure loading aff file %s\n",affpath);
fflush(stderr);
- wordchars = "qwertzuiopasdfghjklyxcvbnmQWERTZUIOPASDFGHJKLYXCVBNM";
+ wordchars = mystrdup("qwertzuiopasdfghjklyxcvbnmQWERTZUIOPASDFGHJKLYXCVBNM");
}
- // Daniel
- derived = NULL;
- sfx = NULL;
- pfx = NULL;
}
@@ -119,6 +121,17 @@ AffixMgr::~AffixMgr()
trystring=NULL;
if (encoding) free(encoding);
encoding=NULL;
+ if (maptable) {
+ for (int j=0; j < nummap; j++) {
+ if (maptable[j].set) free(maptable[j].set);
+ if (maptable[j].set_utf16) free(maptable[j].set_utf16);
+ maptable[j].set = NULL;
+ maptable[j].len = 0;
+ }
+ free(maptable);
+ maptable = NULL;
+ }
+ nummap = 0;
if (reptable) {
for (int j=0; j < numrep; j++) {
free(reptable[j].pattern);
@@ -152,13 +165,6 @@ AffixMgr::~AffixMgr()
if (cpdvowels_utf16) free(cpdvowels_utf16);
if (cpdsyllablenum) free(cpdsyllablenum);
if (utf_tbl) free(utf_tbl);
- if (accent) {
- free(accent->pattern);
- free(accent->replacement);
- accent->pattern = NULL;
- accent->replacement = NULL;
- free(accent);
- }
if (lang) free(lang);
if (wordchars) free(wordchars);
if (version) free(version);
@@ -210,6 +216,10 @@ int AffixMgr::parse_file(const char * affpath)
}
}
+ /* parse COMPLEXPREFIXES for agglutinative languages with right-to-left writing system */
+ if (strncmp(line,"COMPLEXPREFIXES",15) == 0)
+ complexprefixes = 1;
+
/* parse in the flag used by the controlled compound words */
if (strncmp(line,"COMPOUNDFLAG",12) == 0) {
if (parse_flag(line, &compoundflag, "COMPOUNDFLAG")) {
@@ -219,8 +229,14 @@ int AffixMgr::parse_file(const char * affpath)
/* parse in the flag used by compound words */
if (strncmp(line,"COMPOUNDBEGIN",13) == 0) {
- if (parse_flag(line, &compoundbegin, "COMPOUNDBEGIN")) {
- return 1;
+ if (complexprefixes) {
+ if (parse_flag(line, &compoundend, "COMPOUNDBEGIN")) {
+ return 1;
+ }
+ } else {
+ if (parse_flag(line, &compoundbegin, "COMPOUNDBEGIN")) {
+ return 1;
+ }
}
}
@@ -232,8 +248,14 @@ int AffixMgr::parse_file(const char * affpath)
}
/* parse in the flag used by compound words */
if (strncmp(line,"COMPOUNDEND",11) == 0) {
- if (parse_flag(line, &compoundend, "COMPOUNDEND")) {
- return 1;
+ if (complexprefixes) {
+ if (parse_flag(line, &compoundbegin, "COMPOUNDEND")) {
+ return 1;
+ }
+ } else {
+ if (parse_flag(line, &compoundend, "COMPOUNDEND")) {
+ return 1;
+ }
}
}
@@ -321,13 +343,6 @@ int AffixMgr::parse_file(const char * affpath)
}
}
- /* parse in the accent strings */
- if (strncmp(line,"ACCENT",6) == 0) {
- if (parse_accent(line)) {
- return 1;
- }
- }
-
/* parse in the flag used by compound_check() method */
if (strncmp(line,"SYLLABLENUM",11) == 0) {
if (parse_syllablenum(line)) {
@@ -375,20 +390,27 @@ int AffixMgr::parse_file(const char * affpath)
}
}
- /* parse NOMAPSUGS */
- if (strncmp(line,"NOMAPSUGS",11) == 0)
- nomapsugs=1;
+ /* parse MAXNGRAMSUGS */
+ if (strncmp(line,"MAXNGRAMSUGS",12) == 0)
+ maxngramsugs=1;
/* parse NOSPLITSUGS */
if (strncmp(line,"NOSPLITSUGS",11) == 0)
nosplitsugs=1;
+ /* parse SUGSWITHDOTS */
+ if (strncmp(line,"SUGSWITHDOTS",12) == 0)
+ sugswithdots=1;
+
/* parse this affix: P - prefix, S - suffix */
ft = ' ';
- if (strncmp(line,"PFX",3) == 0) ft = 'P';
- if (strncmp(line,"SFX",3) == 0) ft = 'S';
+ if (strncmp(line,"PFX",3) == 0) ft = complexprefixes ? 'S' : 'P';
+ if (strncmp(line,"SFX",3) == 0) ft = complexprefixes ? 'P' : 'S';
if (ft != ' ') {
if (parse_affix(line, ft, afflst)) {
+ fclose(afflst);
+ process_pfx_tree_to_list();
+ process_sfx_tree_to_list();
return 1;
}
}
@@ -396,8 +418,8 @@ int AffixMgr::parse_file(const char * affpath)
}
fclose(afflst);
- // quick loading tree convert to list
-
+ // convert affix trees to sorted list
+ process_pfx_tree_to_list();
process_sfx_tree_to_list();
// now we can speed up performance greatly taking advantage of the
@@ -454,7 +476,11 @@ int AffixMgr::parse_file(const char * affpath)
}
-int AffixMgr::build_pfxlist(AffEntry* pfxptr)
+// we want to be able to quickly access prefix information
+// both by prefix flag, and sorted by prefix string itself
+// so we need to set up two indexes
+
+int AffixMgr::build_pfxtree(AffEntry* pfxptr)
{
PfxEntry * ptr;
PfxEntry * pptr;
@@ -469,7 +495,6 @@ int AffixMgr::build_pfxlist(AffEntry* pfxptr)
ep->setFlgNxt(ptr);
pFlag[flg] = (AffEntry *) ep;
- // next index by affix string
// handle the special case of null affix string
if (strlen(key) == 0) {
@@ -480,29 +505,45 @@ int AffixMgr::build_pfxlist(AffEntry* pfxptr)
return 0;
}
- // now handle the general case
+ // now handle the normal case
+ ep->setNextEQ(NULL);
+ ep->setNextNE(NULL);
+
unsigned char sp = *((const unsigned char *)key);
ptr = (PfxEntry*)pStart[sp];
-
- /* handle the insert at top of list case */
- if ((!ptr) || ( strcmp( ep->getKey() , ptr->getKey() ) <= 0)) {
- ep->setNext(ptr);
+
+ // handle the first insert
+ if (!ptr) {
pStart[sp] = (AffEntry*)ep;
return 0;
}
- /* otherwise find where it fits in order and insert it */
+
+ // otherwise use binary tree insertion so that a sorted
+ // list can easily be generated later
pptr = NULL;
- for (; ptr != NULL; ptr = ptr->getNext()) {
- if (strcmp( ep->getKey() , ptr->getKey() ) <= 0) break;
+ for (;;) {
pptr = ptr;
+ if (strcmp(ep->getKey(), ptr->getKey() ) <= 0) {
+ ptr = ptr->getNextEQ();
+ if (!ptr) {
+ pptr->setNextEQ(ep);
+ break;
+ }
+ } else {
+ ptr = ptr->getNextNE();
+ if (!ptr) {
+ pptr->setNextNE(ep);
+ break;
+ }
+ }
}
- pptr->setNext(ep);
- ep->setNext(ptr);
return 0;
}
-// build an ordered tree of suffix entries (affix reversed)
+// we want to be able to quickly access suffix information
+// both by suffix flag, and sorted by the reverse of the
+// suffix string itself; so we need to set up two indexes
int AffixMgr::build_sfxtree(AffEntry* sfxptr)
{
SfxEntry * ptr;
@@ -518,6 +559,8 @@ int AffixMgr::build_sfxtree(AffEntry* sfxptr)
ep->setFlgNxt(ptr);
sFlag[flg] = (AffEntry *) ep;
+ // next index by affix string
+
// handle the special case of null affix string
if (strlen(key) == 0) {
// always inset them at head of list at element 0
@@ -528,44 +571,83 @@ int AffixMgr::build_sfxtree(AffEntry* sfxptr)
}
// now handle the normal case
- unsigned char sp = *((const unsigned char *)key);
- ptr = (SfxEntry*)sStart[sp];
-
- /* handle the insert at top of tree case */
-
ep->setNextEQ(NULL);
ep->setNextNE(NULL);
+ unsigned char sp = *((const unsigned char *)key);
+ ptr = (SfxEntry*)sStart[sp];
+
+ // handle the first insert
if (!ptr) {
sStart[sp] = (AffEntry*)ep;
return 0;
}
- /* otherwise find where it fits in order and insert it into tree */
-
+ // otherwise use binary tree insertion so that a sorted
+ // list can easily be generated later
pptr = NULL;
for (;;) {
pptr = ptr;
- if (strcmp( ep->getKey(), ptr->getKey() ) <= 0) {
- ptr = ptr->getNextEQ();
- if (!ptr) {
- pptr->setNextEQ(ep);
- break;
- }
+ if (strcmp(ep->getKey(), ptr->getKey() ) <= 0) {
+ ptr = ptr->getNextEQ();
+ if (!ptr) {
+ pptr->setNextEQ(ep);
+ break;
+ }
} else {
- ptr = ptr->getNextNE();
- if (!ptr) {
- pptr->setNextNE(ep);
- break;
- }
+ ptr = ptr->getNextNE();
+ if (!ptr) {
+ pptr->setNextNE(ep);
+ break;
+ }
}
}
return 0;
}
+// convert from binary tree to sorted list
+int AffixMgr::process_pfx_tree_to_list()
+{
+ for (int i=1; i< SETSIZE; i++) {
+ pStart[i] = process_pfx_in_order(pStart[i],NULL);
+ }
+ return 0;
+}
-// initialize the PfxEntry links NextEQ and NextNE to speed searching
+AffEntry* AffixMgr::process_pfx_in_order(AffEntry* ptr, AffEntry* nptr)
+{
+ if (ptr) {
+ nptr = process_pfx_in_order(((PfxEntry*) ptr)->getNextNE(), nptr);
+ ((PfxEntry*) ptr)->setNext((PfxEntry*) nptr);
+ nptr = process_pfx_in_order(((PfxEntry*) ptr)->getNextEQ(), ptr);
+ }
+ return nptr;
+}
+
+
+// convert from binary tree to sorted list
+int AffixMgr:: process_sfx_tree_to_list()
+{
+ for (int i=1; i< SETSIZE; i++) {
+ sStart[i] = process_sfx_in_order(sStart[i],NULL);
+ }
+ return 0;
+}
+
+AffEntry* AffixMgr::process_sfx_in_order(AffEntry* ptr, AffEntry* nptr)
+{
+ if (ptr) {
+ nptr = process_sfx_in_order(((SfxEntry*) ptr)->getNextNE(), nptr);
+ ((SfxEntry*) ptr)->setNext((SfxEntry*) nptr);
+ nptr = process_sfx_in_order(((SfxEntry*) ptr)->getNextEQ(), ptr);
+ }
+ return nptr;
+}
+
+
+// reinitialize the PfxEntry links NextEQ and NextNE to speed searching
+// using the idea of leading subsets this time
int AffixMgr::process_pfx_order()
{
PfxEntry* ptr;
@@ -613,26 +695,8 @@ int AffixMgr::process_pfx_order()
return 0;
}
-// convert SfxEntry tree to list with inorder tree walking
-AffEntry * AffixMgr::process_sfx_inorder(AffEntry* ptr, AffEntry* nptr) {
- if (ptr) {
- nptr = process_sfx_inorder(((SfxEntry*) ptr)->getNextNE(), nptr);
- ((SfxEntry*) ptr)->setNext((SfxEntry*) nptr);
- nptr = process_sfx_inorder(((SfxEntry*) ptr)->getNextEQ(), ptr);
- }
- return nptr;
-}
-
-// convert SfxEntry trees to lists with inorder tree walking
-int AffixMgr::process_sfx_tree_to_list()
-{
- for (int i=1; i < SETSIZE; i++) {
- sStart[i] = process_sfx_inorder(sStart[i], NULL);
- }
- return 0;
-}
-
// initialize the SfxEntry links NextEQ and NextNE to speed searching
+// using the idea of leading subsets this time
int AffixMgr::process_sfx_order()
{
SfxEntry* ptr;
@@ -692,7 +756,7 @@ void AffixMgr::encodeit(struct affentry * ptr, char * cs)
unsigned char c;
int i, j, k;
unsigned char mbr[MAXLNLEN];
- w_char wmbr[MAXLNLEN/2];
+ w_char wmbr[MAXLNLEN];
w_char * wpos = wmbr;
// now clear the conditions array */
@@ -788,7 +852,7 @@ void AffixMgr::encodeit(struct affentry * ptr, char * cs)
if (k >> 7) {
u8_u16(wpos, 1, (char *) mbr + j);
wpos++;
- if ((k & 0xe0) == 0xe0) j+=2; else j++; // 3-byte UTF-8 character?
+ if ((k & 0xe0) == 0xe0) j+=2; else j++; // 3-byte UTF-8 character
} else {
ptr->conds.utf8.ascii[k] = ptr->conds.utf8.ascii[k] | (1 << n);
}
@@ -801,7 +865,7 @@ void AffixMgr::encodeit(struct affentry * ptr, char * cs)
if (k >> 7) {
u8_u16(wpos, 1, (char *) mbr + j);
wpos++;
- if ((k & 0xe0) == 0xe0) j+=2; else j++; // 3-byte UTF-8 character?
+ if ((k & 0xe0) == 0xe0) j+=2; else j++; // 3-byte UTF-8 character
} else {
ptr->conds.utf8.ascii[k] = ptr->conds.utf8.ascii[k] & ~(1 << n);
}
@@ -823,7 +887,7 @@ void AffixMgr::encodeit(struct affentry * ptr, char * cs)
ptr->conds.utf8.wchars[n] = (w_char *) malloc(sizeof(w_char));
ptr->conds.utf8.wlen[n] = 1;
u8_u16(ptr->conds.utf8.wchars[n], 1, cs + i);
- if ((c & 0xe0) == 0xe0) i+=2; else i++; // BUG???
+ if ((c & 0xe0) == 0xe0) i+=2; else i++; // 3-byte UFT-8 character
} else {
ptr->conds.utf8.wchars[n] = NULL;
// not a group so just set the proper bit for this char
@@ -1307,7 +1371,7 @@ struct hentry * AffixMgr::compound_check(const char * word, int len,
((oldwordnum > 0) && compoundmiddle && TESTAFF(rv->astr, compoundmiddle, rv->alen))
//MAGYARISPELL
|| ((langnum == LANG_hu) && hu_mov_rule && (
- TESTAFF(rv->astr, 'F', rv->alen) || // XXX wired Magyar Ispell codes
+ TESTAFF(rv->astr, 'F', rv->alen) || // XXX hardwired Magyar Ispell codes
TESTAFF(rv->astr, 'G', rv->alen) ||
TESTAFF(rv->astr, 'H', rv->alen)
)
@@ -2304,8 +2368,8 @@ char * AffixMgr::affix_check_morph(const char * word, int len, const FLAG needfl
}
-int AffixMgr::expand_rootword(struct guessword * wlst, int maxn,
- const char * ts, int wl, const unsigned short * ap, unsigned short al)
+int AffixMgr::expand_rootword(struct guessword * wlst, int maxn, const char * ts,
+ int wl, const unsigned short * ap, unsigned short al, char * bad, int badl)
{
int nh=0;
@@ -2323,16 +2387,19 @@ int AffixMgr::expand_rootword(struct guessword * wlst, int maxn,
unsigned short c = (unsigned short) ap[i];
SfxEntry * sptr = (SfxEntry *)sFlag[c];
while (sptr) {
- char * newword = sptr->add(ts, wl);
- if (newword) {
- if (nh < maxn) {
- wlst[nh].word = newword;
- wlst[nh].allow = sptr->allowCross();
- nh++;
- } else {
- free(newword);
- }
- }
+ if (!sptr->getKeyLen() || ((badl > sptr->getKeyLen()) &&
+ (strcmp(sptr->getAffix(), bad + badl - sptr->getKeyLen()) == 0))) {
+ char * newword = sptr->add(ts, wl);
+ if (newword) {
+ if (nh < maxn) {
+ wlst[nh].word = newword;
+ wlst[nh].allow = sptr->allowCross();
+ nh++;
+ } else {
+ free(newword);
+ }
+ }
+ }
sptr = (SfxEntry *)sptr ->getFlgNxt();
}
}
@@ -2346,7 +2413,8 @@ int AffixMgr::expand_rootword(struct guessword * wlst, int maxn,
unsigned short c = (unsigned short) ap[k];
PfxEntry * cptr = (PfxEntry *) pFlag[c];
while (cptr) {
- if (cptr->allowCross()) {
+ if (cptr->allowCross() && (!cptr->getKeyLen() || ((badl > cptr->getKeyLen()) &&
+ (strncmp(cptr->getKey(), bad, cptr->getKeyLen()) == 0)))) {
int l1 = strlen(wlst[j].word);
char * newword = cptr->add(wlst[j].word, l1);
if (newword) {
@@ -2370,16 +2438,19 @@ int AffixMgr::expand_rootword(struct guessword * wlst, int maxn,
unsigned short c = (unsigned short) ap[m];
PfxEntry * ptr = (PfxEntry *) pFlag[c];
while (ptr) {
- char * newword = ptr->add(ts, wl);
- if (newword) {
- if (nh < maxn) {
- wlst[nh].word = newword;
- wlst[nh].allow = ptr->allowCross();
- nh++;
- } else {
- free(newword);
- }
- }
+ if (!ptr->getKeyLen() || ((badl > ptr->getKeyLen()) &&
+ (strncmp(ptr->getKey(), bad, ptr->getKeyLen()) == 0))) {
+ char * newword = ptr->add(ts, wl);
+ if (newword) {
+ if (nh < maxn) {
+ wlst[nh].word = newword;
+ wlst[nh].allow = ptr->allowCross();
+ nh++;
+ } else {
+ free(newword);
+ }
+ }
+ }
ptr = (PfxEntry *)ptr ->getFlgNxt();
}
}
@@ -2415,13 +2486,6 @@ struct mapentry * AffixMgr::get_maptable()
return maptable;
}
-// return replacing table
-struct replentry * AffixMgr::get_accent()
-{
- if (! accent ) return NULL;
- return accent;
-}
-
// return text encoding of dictionary
char * AffixMgr::get_encoding()
{
@@ -2443,6 +2507,12 @@ struct unicode_info2 * AffixMgr::get_utf_conv()
return utf_tbl;
}
+// return complexprefixes
+int AffixMgr::get_complexprefixes()
+{
+ return complexprefixes;
+}
+
// return the preferred try string for suggestions
char * AffixMgr::get_try_string()
{
@@ -2543,15 +2613,15 @@ const int AffixMgr::have_contclass()
}
// return utf8
-int AffixMgr::is_utf8()
+int AffixMgr::get_utf8()
{
return utf8;
}
// return nosplitsugs
-int AffixMgr::get_nomapsugs(void)
+int AffixMgr::get_maxngramsugs(void)
{
- return nomapsugs;
+ return maxngramsugs;
}
// return nosplitsugs
@@ -2560,6 +2630,12 @@ int AffixMgr::get_nosplitsugs(void)
return nosplitsugs;
}
+// return sugswithdots
+int AffixMgr::get_sugswithdots(void)
+{
+ return sugswithdots;
+}
+
/* parse in the try string */
int AffixMgr::parse_try(char * line)
{
@@ -2702,45 +2778,6 @@ int AffixMgr::parse_num(char * line, int * out, char * name)
return 0;
}
-/* parse in the accent string for suggestaccent method */
-int AffixMgr::parse_accent(char * line)
-{
- w_char acc[MAXACC];
- if (accent) {
- fprintf(stderr,"error: duplicate accent strings used\n");
- return 1;
- }
- char * tp = line;
- char * piece;
- int i = 0;
- int np = 0;
- while ((piece=mystrsep(&tp,' '))) {
- if (*piece != '\0') {
- switch(i) {
- case 0: { np++; break; }
- case 1: {
- accent = (replentry *) malloc(sizeof(struct replentry));
- accent->pattern = mystrdup(piece); np++; break; }
- case 2: { accent->replacement = mystrdup(piece); np++; break; }
- default: break;
- }
- i++;
- }
- free(piece);
- }
- if (np < 2) {
- fprintf(stderr,"error: missing compound word information\n");
- return 1;
- } else if ((utf8 && (u8_u16(acc, MAXACC, accent->pattern) !=
- u8_u16(acc, MAXACC, accent->replacement))) ||
- (!utf8 && (strlen(accent->pattern) !=
- strlen(accent->replacement)))) {
- fprintf(stderr,"error: the length of two accent strings not equal\n");
- return 1;
- }
- return 0;
-}
-
/* parse in the wordchars string */
int AffixMgr::parse_wordchars(char * line)
{
@@ -2809,7 +2846,7 @@ int AffixMgr::parse_cpdsyllable(char * line)
fprintf(stderr,"error: missing compoundsyllable information\n");
return 1;
}
- if (np == 2) cpdvowels = "aeiouAEIOU";
+ if (np == 2) cpdvowels = mystrdup("aeiouAEIOU");
return 0;
}
@@ -2968,8 +3005,23 @@ int AffixMgr::parse_maptable(char * line, FILE * af)
}
break;
}
- case 1: { maptable[j].set = mystrdup(piece);
- maptable[j].len = strlen(maptable[j].set);
+ case 1: {
+ maptable[j].len = 0;
+ maptable[j].set = NULL;
+ maptable[j].set_utf16 = NULL;
+ if (!utf8) {
+ maptable[j].set = mystrdup(piece);
+ maptable[j].len = strlen(maptable[j].set);
+ } else {
+ w_char w[MAXWORDLEN];
+ int n = u8_u16(w, MAXWORDLEN, piece);
+ if (n > 0) {
+ flag_qsort((unsigned short *) w, 0, n - 1);
+ maptable[j].set_utf16 = (w_char *) malloc(n * sizeof(w_char));
+ memcpy(maptable[j].set_utf16, w, n * sizeof(w_char));
+ }
+ maptable[j].len = n;
+ }
break; }
default: break;
}
@@ -2977,7 +3029,7 @@ int AffixMgr::parse_maptable(char * line, FILE * af)
}
free(piece);
}
- if ((!(maptable[j].set)) || (!(maptable[j].len))) {
+ if ((!(maptable[j].set || maptable[j].set_utf16)) || (!(maptable[j].len))) {
fprintf(stderr,"error: map table is corrupt\n");
return 1;
}
@@ -3048,6 +3100,9 @@ int AffixMgr::parse_affix(char * line, const char at, FILE * af)
char * piece;
int i = 0;
+ // checking lines with bad syntax
+ int basefieldnum = 0;
+
// split affix header line into pieces
int np = 0;
@@ -3133,6 +3188,9 @@ int AffixMgr::parse_affix(char * line, const char at, FILE * af)
// piece 3 - is string to strip or 0 for null
case 2: {
np++;
+ if (complexprefixes) {
+ if (utf8) reverseword_utf(piece); else reverseword(piece);
+ }
nptr->strip = mystrdup(piece);
nptr->stripl = strlen(nptr->strip);
if (strcmp(nptr->strip,"0") == 0) {
@@ -3153,6 +3211,9 @@ int AffixMgr::parse_affix(char * line, const char at, FILE * af)
dash = strchr(piece, '/');
if (dash) {
*dash = '\0';
+ if (complexprefixes) {
+ if (utf8) reverseword_utf(piece); else reverseword(piece);
+ }
nptr->appnd = mystrdup(piece);
nptr->contclasslen = pHMgr->decode_flags(&(nptr->contclass), dash + 1);
flag_qsort(nptr->contclass, 0, nptr->contclasslen);
@@ -3163,6 +3224,9 @@ int AffixMgr::parse_affix(char * line, const char at, FILE * af)
contclasses[(nptr->contclass)[i]] = 1;
}
} else {
+ if (complexprefixes) {
+ if (utf8) reverseword_utf(piece); else reverseword(piece);
+ }
nptr->appnd = mystrdup(piece);
}
@@ -3178,18 +3242,50 @@ int AffixMgr::parse_affix(char * line, const char at, FILE * af)
// piece 5 - is the conditions descriptions
case 4: {
np++;
+ if (complexprefixes) {
+ int neg = 0;
+ if (utf8) reverseword_utf(piece); else reverseword(piece);
+ // reverse condition
+ for (char * k = piece + strlen(piece) - 1; k >= piece; k--) {
+ switch(*k) {
+ case '[': {
+ if (neg) *(k+1) = '['; else *k = ']';
+ break;
+ }
+ case ']': {
+ *k = '[';
+ if (neg) *(k+1) = '^';
+ neg = 0;
+ break;
+ }
+ case '^': {
+ if (*(k+1) == ']') neg = 1; else *(k+1) = *k;
+ break;
+ }
+ default: {
+ if (neg) *(k+1) = *k;
+ }
+ }
+ }
+ }
+ if (nptr->stripl && (strcmp(piece, ".") != 0) &&
+ redundant_condition(at, nptr->strip, nptr->stripl, piece, nl))
+ strcpy(piece, ".");
encodeit(nptr,piece);
break;
}
- // piece 6, 7 - continue classes, and morph. description
case 5: {
np++;
+ if (complexprefixes) {
+ if (utf8) reverseword_utf(piece); else reverseword(piece);
+ }
nptr->morphcode = mystrdup(piece);
break;
}
case 6: {
+ // XXX deprecated syntax
np++;
if (nptr->contclass) {
fprintf(stderr, "error: affix rule contains two contclass "
@@ -3215,11 +3311,20 @@ int AffixMgr::parse_affix(char * line, const char at, FILE * af)
// check to make sure we parsed enough pieces
if (np < 5) { // HUNMORPH
char * err = pHMgr->encode_flag(aflag);
- fprintf(stderr, "error: affix %s is corrupt near line %s\n", aflag, nl);
+ fprintf(stderr, "error: affix %s is corrupt near line %s\n", err, nl);
free(err);
free(ptr);
return 1;
}
+
+ // detect unnecessary fields, excepting comments
+ if (basefieldnum) {
+ int fieldnum = !(nptr->morphcode) ? 5 : ((*(nptr->morphcode)=='#') ? 5 : 6);
+ if (fieldnum != basefieldnum)
+ fprintf(stderr, "warning - bad field number:\n%s\n", nl);
+ } else {
+ basefieldnum = !(nptr->morphcode) ? 5 : ((*(nptr->morphcode)=='#') ? 5 : 6);
+ }
nptr++;
}
@@ -3229,7 +3334,7 @@ int AffixMgr::parse_affix(char * line, const char at, FILE * af)
for (int k = 0; k < numents; k++) {
if (at == 'P') {
PfxEntry * pfxptr = new PfxEntry(this,nptr);
- build_pfxlist((AffEntry *)pfxptr);
+ build_pfxtree((AffEntry *)pfxptr);
} else {
SfxEntry * sfxptr = new SfxEntry(this,nptr);
build_sfxtree((AffEntry *)sfxptr);
@@ -3251,3 +3356,69 @@ void AffixMgr::set_spec_utf8_encoding() {
}
}
}
+
+int AffixMgr::redundant_condition(char ft, char * strip, int stripl, const char * cond, char * line) {
+ int condl = strlen(cond);
+ int i;
+ int j;
+ int neg;
+ int in;
+ if (ft == 'P') { // prefix
+ if (strncmp(strip, cond, condl) == 0) return 1;
+ if (utf8) {
+ } else {
+ for (i = 0, j = 0; (i < stripl) && (j < condl); i++, j++) {
+ if (cond[j] != '[') {
+ if (cond[j] != strip[i]) {
+ fprintf(stderr, "warning - incompatible stripping characters and condition:\n%s\n", line);
+ }
+ } else {
+ neg = (cond[j+1] == '^') ? 1 : 0;
+ in = 0;
+ do {
+ j++;
+ if (strip[i] == cond[j]) in = 1;
+ } while ((j < (condl - 1)) && (cond[j] != ']'));
+ if (j == (condl - 1) && (cond[j] != ']')) {
+ fprintf(stderr, "error - missing ] in condition:\n%s\n", line);
+ return 0;
+ }
+ if ((!neg && !in) || (neg && in)) {
+ fprintf(stderr, "warning - incompatible stripping characters and condition:\n%s\n", line);
+ return 0;
+ }
+ }
+ }
+ if (j >= condl) return 1;
+ }
+ } else { // suffix
+ if ((stripl >= condl) && strcmp(strip + stripl - condl, cond) == 0) return 1;
+ if (utf8) {
+ } else {
+ for (i = stripl - 1, j = condl - 1; (i >= 0) && (j >= 0); i--, j--) {
+ if (cond[j] != ']') {
+ if (cond[j] != strip[i]) {
+ fprintf(stderr, "warning - incompatible stripping characters and condition:\n%s\n", line);
+ }
+ } else {
+ in = 0;
+ do {
+ j--;
+ if (strip[i] == cond[j]) in = 1;
+ } while ((j > 0) && (cond[j] != '['));
+ if ((j == 0) && (cond[j] != '[')) {
+ fprintf(stderr, "error - missing ] in condition:\n%s\n", line);
+ return 0;
+ }
+ neg = (cond[j+1] == '^') ? 1 : 0;
+ if ((!neg && !in) || (neg && in)) {
+ fprintf(stderr, "warning - incompatible stripping characters and condition:\n%s\n", line);
+ return 0;
+ }
+ }
+ }
+ if (j < 0) return 1;
+ }
+ }
+ return 0;
+}
diff --git a/src/hunspell/affixmgr.hxx b/src/hunspell/affixmgr.hxx
index 690b798..180ff19 100644
--- a/src/hunspell/affixmgr.hxx
+++ b/src/hunspell/affixmgr.hxx
@@ -20,6 +20,7 @@ class AffixMgr
char * encoding;
int utf8;
struct unicode_info2 * utf_tbl;
+ int complexprefixes;
FLAG compoundflag;
FLAG compoundbegin;
FLAG compoundmiddle;
@@ -35,14 +36,14 @@ class AffixMgr
replentry * reptable;
int nummap;
mapentry * maptable;
- int nomapsugs;
+ int maxngramsugs;
int nosplitsugs;
+ int sugswithdots;
int cpdwordmax;
int cpdmaxsyllable;
char * cpdvowels;
w_char * cpdvowels_utf16;
int cpdvowels_utf16_len;
- replentry * accent;
char * cpdsyllablenum;
const char * pfxappnd; // BUG: not stateless
const char * sfxappnd; // BUG: not stateless
@@ -93,8 +94,8 @@ public:
char * suffix_check_twosfx_morph(const char * word, int len,
int sfxopts, AffEntry * ppfx, const FLAG needflag = FLAG_NULL);
- int expand_rootword(struct guessword * wlst, int maxn,
- const char * ts, int wl, const unsigned short * ap, unsigned short al);
+ int expand_rootword(struct guessword * wlst, int maxn, const char * ts,
+ int wl, const unsigned short * ap, unsigned short al, char * bad, int);
int get_syllable (const char * word, int wlen);
int repl_check(const char * word, int len);
@@ -111,7 +112,6 @@ public:
struct hentry * lookup(const char * word);
int get_numrep();
struct replentry * get_reptable();
- struct replentry * get_accent();
int get_nummap();
struct mapentry * get_maptable();
char * get_encoding();
@@ -133,10 +133,12 @@ public:
const char * get_derived();
const char * get_version();
const int have_contclass();
- int is_utf8();
+ int get_utf8();
+ int get_complexprefixes();
char * get_suffixed(char );
- int get_nomapsugs();
+ int get_maxngramsugs();
int get_nosplitsugs();
+ int get_sugswithdots(void);
private:
int parse_file(const char * affpath);
@@ -149,7 +151,6 @@ private:
int parse_forbid(char * line);
int parse_onlyroot(char * line);
int parse_cpdsyllable(char * line);
- int parse_accent(char * line);
int parse_syllablenum(char * line);
int parse_reptable(char * line, FILE * af);
int parse_maptable(char * line, FILE * af);
@@ -159,14 +160,16 @@ private:
int parse_version(char * line);
void encodeit(struct affentry * ptr, char * cs);
- int build_pfxlist(AffEntry* pfxptr);
+ int build_pfxtree(AffEntry* pfxptr);
int build_sfxtree(AffEntry* sfxptr);
int process_pfx_order();
int process_sfx_order();
- AffEntry * process_sfx_inorder(AffEntry * ptr, AffEntry * nptr);
+ AffEntry * process_pfx_in_order(AffEntry * ptr, AffEntry * nptr);
+ AffEntry * process_sfx_in_order(AffEntry * ptr, AffEntry * nptr);
+ int process_pfx_tree_to_list();
int process_sfx_tree_to_list();
void set_spec_utf8_encoding();
-
+ int redundant_condition(char, char * strip, int stripl, const char * cond, char *);
};
#endif
diff --git a/src/hunspell/atypes.hxx b/src/hunspell/atypes.hxx
index 6ae87e9..e9eb876 100644
--- a/src/hunspell/atypes.hxx
+++ b/src/hunspell/atypes.hxx
@@ -7,8 +7,6 @@
#include "csutil.hxx"
#include "hashmgr.hxx"
-extern int flag_bsearch(unsigned char flags[], unsigned char flag, short right);
-
#define SETSIZE 256
#define CONTSIZE 65536
#define MAXWORDLEN 300
@@ -62,6 +60,7 @@ struct replentry {
struct mapentry {
char * set;
+ w_char * set_utf16;
int len;
};
diff --git a/src/hunspell/csutil.cxx b/src/hunspell/csutil.cxx
index 66e74b8..5b2c3d2 100644
--- a/src/hunspell/csutil.cxx
+++ b/src/hunspell/csutil.cxx
@@ -9,8 +9,6 @@
#include "utf_info.cxx"
#define UTF_LST_LEN (sizeof(utf_lst) / (sizeof(unicode_info)))
-#include "config.h"
-
#ifndef W32
using namespace std;
#endif
@@ -432,24 +430,51 @@ char * mystrrep(char * word, const char * pat, const char * rep) {
void mkallsmall(char * p, const struct cs_info * csconv)
{
while (*p != '\0') {
-
*p = csconv[((unsigned char) *p)].clower;
p++;
}
}
-
+
+void mkallsmall_utf(w_char * u, int nc, struct unicode_info2 * utfconv) {
+ for (int i = 0; i < nc; i++) {
+ unsigned short idx = (u[i].h << 8) + u[i].l;
+ if (idx != utfconv[idx].clower) {
+ u[i].h = (unsigned char) (utfconv[idx].clower >> 8);
+ u[i].l = (unsigned char) (utfconv[idx].clower & 0x00FF);
+ }
+ }
+}
// convert null terminated string to have intial capital
void mkinitcap(char * p, const struct cs_info * csconv)
{
if (*p != '\0') *p = csconv[((unsigned char)*p)].cupper;
}
-
-
-
-
-
+ // reverse word
+ void reverseword(char * word) {
+ char r;
+ for (char * dest = word + strlen(word) - 1; word < dest; word++, dest--) {
+ r=*word;
+ *word = *dest;
+ *dest = r;
+ }
+ }
+
+ // reverse word
+ void reverseword_utf(char * word) {
+ w_char w[MAXWORDLEN];
+ w_char * p;
+ w_char r;
+ int l = u8_u16(w, MAXWORDLEN, word);
+ p = w;
+ for (w_char * dest = w + l - 1; p < dest; p++, dest--) {
+ r=*p;
+ *p = *dest;
+ *dest = r;
+ }
+ u16_u8(word, MAXWORDLEN, w, l);
+ }
// these are simple character mappings for the
// encodings supported by MySpell
@@ -4428,7 +4453,3 @@ int get_lang_num(const char * lang) {
}
return LANG_xx;
};
-
-const char * hunspell_version() {
- return VERSION;
-}
diff --git a/src/hunspell/csutil.hxx b/src/hunspell/csutil.hxx
index 31b409e..2a4d4cf 100644
--- a/src/hunspell/csutil.hxx
+++ b/src/hunspell/csutil.hxx
@@ -56,6 +56,12 @@ void strlinecat(char * lines, const char * s);
// leave only last {[^}]*} pattern in string
char * delete_zeros(char * morphout);
+// reverse word
+ void reverseword(char *);
+
+// reverse word
+ void reverseword_utf(char *);
+
// character encoding information
struct cs_info {
unsigned char ccase;
@@ -117,7 +123,7 @@ void mkallsmall(char * p, const struct cs_info * csconv);
// convert null terminated string to have intial capital
void mkinitcap(char * p, const struct cs_info * csconv);
-// get library version
-const char * hunspell_version();
+// convert first nc characters of UTF-8 string to little
+void mkallsmall_utf(w_char * u, int nc, struct unicode_info2 * utfconv);
#endif
diff --git a/src/hunspell/hashmgr.cxx b/src/hunspell/hashmgr.cxx
index 83d5d68..8b176d7 100644
--- a/src/hunspell/hashmgr.cxx
+++ b/src/hunspell/hashmgr.cxx
@@ -1,10 +1,6 @@
#include <license.hunspell>
#include <license.myspell>
-#ifndef W32
-#include <unistd.h>
-#endif
-
#include <cstdlib>
#include <cstring>
#ifdef HAVE_FCNTL_H
@@ -16,10 +12,10 @@
#include "csutil.hxx"
#ifndef W32
+#include <unistd.h>
using namespace std;
#endif
-
// build a hash table from a munched word list
HashMgr::HashMgr(const char * tpath, const char * apath)
@@ -27,6 +23,8 @@ HashMgr::HashMgr(const char * tpath, const char * apath)
tablesize = 0;
tableptr = NULL;
flag_mode = FLAG_CHAR;
+ complexprefixes = 0;
+ utf8 = 0;
load_config(apath);
int ec = load_tables(tpath);
if (ec) {
@@ -50,15 +48,16 @@ HashMgr::~HashMgr()
struct hentry * pt = &tableptr[i];
struct hentry * nt = NULL;
if (pt) {
- if (pt->word) free(pt->word);
if (pt->astr) free(pt->astr);
+ if (pt->word) free(pt->word);
if (pt->description) free(pt->description);
+
pt = pt->next;
}
while(pt) {
nt = pt->next;
- if (pt->word) free(pt->word);
if (pt->astr) free(pt->astr);
+ if (pt->word) free(pt->word);
if (pt->description) free(pt->description);
free(pt);
pt = nt;
@@ -93,12 +92,15 @@ int HashMgr::add_word(const char * word, int wl, unsigned short * aff, int al, c
hp->wlen = wl;
hp->alen = al;
hp->word = mystrdup(word);
+ if (complexprefixes) {
+ if (utf8) reverseword_utf(hp->word); else reverseword(hp->word);
+ }
hp->astr = aff;
hp->next = NULL;
hp->next_homonym = NULL;
hp->description = mystrdup(desc);
- int i = hash(word);
+ int i = hash(hp->word);
struct hentry * dp = &tableptr[i];
if (dp->word == NULL) {
@@ -184,6 +186,7 @@ int HashMgr::load_tables(const char * tpath)
char ts[MAXDELEN];
if (! fgets(ts, MAXDELEN-1,rawdict)) return 2;
mychomp(ts);
+ if ((*ts < '1') || (*ts > '9')) fprintf(stderr, "error - missing word count in dictionary file\n");
tablesize = atoi(ts);
tablesize = tablesize + 5 + USERWORD;
if ((tablesize %2) == 0) tablesize++;
@@ -352,13 +355,15 @@ int HashMgr::load_config(const char * affpath)
if (flag_mode != FLAG_CHAR) {
fprintf(stderr,"error: duplicate FLAG parameter\n");
}
- if (strstr(line, "long")) flag_mode = FLAG_LONG;
- if (strstr(line, "num")) flag_mode = FLAG_NUM;
+ if (strstr(line, " long")) flag_mode = FLAG_LONG;
+ if (strstr(line, " num")) flag_mode = FLAG_NUM;
if (flag_mode == FLAG_CHAR) {
fprintf(stderr,"error: FLAG need `num' or `long' parameter: %s\n", line);
}
}
- if ((strncmp(line,"SFX ",4) == 0) || (strncmp(line,"PFX ",4) == 0)) break;
+ if ((strncmp(line,"SET ",4) == 0) && strstr(line, " UTF-8")) utf8 = 1;
+ if (strncmp(line,"COMPLEXPREFIXES",15) == 0) complexprefixes = 1;
+ if ((strncmp(line,"SFX ",4) == 0) || (strncmp(line,"PFX ",4) == 0)) break;
}
fclose(afflst);
return 0;
diff --git a/src/hunspell/hashmgr.hxx b/src/hunspell/hashmgr.hxx
index e7e3002..4e60094 100644
--- a/src/hunspell/hashmgr.hxx
+++ b/src/hunspell/hashmgr.hxx
@@ -11,6 +11,8 @@ class HashMgr
struct hentry * tableptr;
int userword;
flag flag_mode;
+ int complexprefixes;
+ int utf8;
public:
HashMgr(const char * tpath, const char * apath);
diff --git a/src/hunspell/hunspell.cxx b/src/hunspell/hunspell.cxx
index 4c9bf28..125bcd3 100644
--- a/src/hunspell/hunspell.cxx
+++ b/src/hunspell/hunspell.cxx
@@ -1,10 +1,6 @@
#include <license.hunspell>
#include <license.myspell>
-#ifdef HAVE_CONFIG_H
-#include <config.h>
-#endif
-
#include <cstring>
#include <cstdlib>
#include <cstdio>
@@ -21,6 +17,7 @@ Hunspell::Hunspell(const char * affpath, const char * dpath)
csconv = NULL;
utfconv = NULL;
utf8 = 0;
+ complexprefixes = 0;
/* first set up the hash manager */
pHMgr = new HashMgr(dpath, affpath);
@@ -35,11 +32,10 @@ Hunspell::Hunspell(const char * affpath, const char * dpath)
encoding = pAMgr->get_encoding();
csconv = get_current_cs(encoding);
langnum = pAMgr->get_langnum();
- if (pAMgr->is_utf8()) {
- utfconv = pAMgr->get_utf_conv();
- utf8 = 1;
- }
-
+ utf8 = pAMgr->get_utf8();
+ utfconv = pAMgr->get_utf_conv();
+ complexprefixes = pAMgr->get_complexprefixes();
+
/* and finally set up the suggestion manager */
pSMgr = new SuggestMgr(try_string, MAXSUGGESTION, pAMgr);
if (try_string) free(try_string);
@@ -436,8 +432,8 @@ int Hunspell::spell(const char * word)
if (rv) return 1;
- // LANG_hu, LANG_de section: compoundings with dashes
- if ((langnum == LANG_hu) or (langnum == LANG_de)) {
+ // LANG_hu: compoundings with dashes XXX deprecated!
+ if (langnum == LANG_hu) {
int n;
// compound word with dash (HU) I18n
char * dash;
@@ -480,30 +476,46 @@ int Hunspell::spell(const char * word)
return 0;
}
-struct hentry * Hunspell::check(const char * word)
+struct hentry * Hunspell::check(const char * w)
{
struct hentry * he = NULL;
int len;
+ char w2[MAXWORDLEN];
+ const char * word = w;
+
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ strcpy(w2, w);
+ if (utf8) reverseword_utf(w2); else reverseword(w2);
+ word = w2;
+ }
+
+ forbidden_compound = 0; // XXX LANG_hu class variable for suggestions (not threadsafe)
+ prevcompound = 0; // compounding information for Hunspell's pipe interface (not threadsafe)
+ prevroot = NULL; // root information for Hunspell's pipe interface (not threadsafe)
- forbidden_compound = 0;
- prevcompound = 0;
- prevroot = NULL;
-
- if (pHMgr)
- he = pHMgr->lookup(word);
+ // look word in hash table
+ if (pHMgr) he = pHMgr->lookup(word);
- // BUG: homonyms, not threadsafe
- // check forbidden words
+ // check forbidden words (FIXME: don't check forbiddenword homonyms)
if ((he) && (he->astr) && (pAMgr) && TESTAFF(he->astr, pAMgr->get_forbiddenword(), he->alen)) {
+ // LANG_hu section: set dash information for suggestions
+ if (langnum == LANG_hu) {
forbidden_compound = 1;
if (pAMgr->get_compoundflag() &&
TESTAFF(he->astr, pAMgr->get_compoundflag(), he->alen)) {
forbidden_compound = 2;
}
- return NULL;
+ }
+ return NULL;
}
- if (((he == NULL) || TESTAFF(he->astr, pAMgr->get_pseudoroot(), he->alen)) && (pAMgr)) {
+ // he = next not pseudoroot homonym or NULL
+ while (he && (he->astr) && TESTAFF(he->astr, pAMgr->get_pseudoroot(), he->alen))
+ he = he->next_homonym;
+
+ // check with affixes
+ if (!he && pAMgr) {
// try stripping off affixes */
len = strlen(word);
he = pAMgr->affix_check(word, len, 0);
@@ -513,14 +525,14 @@ struct hentry * Hunspell::check(const char * word)
if ((he->astr) && (pAMgr) &&
TESTAFF(he->astr, pAMgr->get_forbiddenword(), he->alen)
&& (! TESTAFF(he->astr, pAMgr->get_onlyroot(), he->alen))) {
- forbidden_compound = 1;
+ forbidden_compound = 1; // LANG_hu
return NULL;
}
prevroot = he->word;
} else if (pAMgr->get_compoundflag() || pAMgr->get_compoundbegin()) {
he = pAMgr->compound_check(word, len,
0,0,100,0,NULL,NULL);
- // LANG_hu section: `moving rule' with last dash
+ // LANG_hu section: `moving rule' with last dash
if ((!he) && (langnum == LANG_hu) && (word[len-1]=='-')) {
char * dup = mystrdup(word);
dup[len-1] = '\0';
@@ -528,18 +540,16 @@ struct hentry * Hunspell::check(const char * word)
-5,0,100,1,NULL,NULL);
free(dup);
}
- // end of LANG speficic region
+ // end of LANG speficic region
if (he) {
- prevroot = he->word; // XXX
+ prevroot = he->word;
prevcompound = 1;
}
}
}
- if (he) return he;
-
- return NULL;
+ return he;
}
int Hunspell::suggest(char*** slst, const char * word)
@@ -555,8 +565,9 @@ int Hunspell::suggest(char*** slst, const char * word)
int wl = cleanword2(cw, word, unicw, &nc, &captype, &abbv);
if (wl == 0) return 0;
int ns = 0;
- *slst = NULL; // HU, nsug in pSMgr->suggest
-
+ *slst = NULL;
+ int capwords = 0;
+
switch(captype) {
case NOCAP: {
ns = pSMgr->suggest(slst, cw, ns);
@@ -568,9 +579,12 @@ int Hunspell::suggest(char*** slst, const char * word)
memcpy(wspace,cw,(wl+1));
mkallsmall2(wspace, unicw, nc);
ns = pSMgr->suggest(slst, wspace, ns);
- for (int j=0; j < ns; j++)
- mkinitcap((*slst)[j]);
- ns = pSMgr->suggest(slst, cw, ns);
+
+// for (int j=0; j < ns; j++)
+// mkinitcap((*slst)[j]);
+ capwords = 1;
+
+ ns = pSMgr->suggest(slst, cw, ns);
break;
}
@@ -624,32 +638,61 @@ int Hunspell::suggest(char*** slst, const char * word)
// try ngram approach since found nothing
if (ns == 0) {
- ns = pSMgr->ngsuggest(*slst, cw, pHMgr);
- if (ns) {
- switch(captype) {
- case NOCAP: break;
- case HUHCAP: break;
- case INITCAP: {
- for (int j=0; j < ns; j++)
- mkinitcap((*slst)[j]);
- }
- break;
-
- case ALLCAP: {
- for (int j=0; j < ns; j++)
- mkallcap((*slst)[j]);
- }
- break;
+ switch(captype) {
+ case NOCAP:
+ case HUHCAP: {
+ ns = pSMgr->ngsuggest(*slst, cw, pHMgr);
+ break;
+ }
+ case INITCAP: {
+ memcpy(wspace,cw,(wl+1));
+ mkallsmall2(wspace, unicw, nc);
+ ns = pSMgr->ngsuggest(*slst, wspace, pHMgr);
+ capwords = 1;
+
+// for (int j=0; j < ns; j++)
+// mkinitcap((*slst)[j]);
+ break;
+ }
+ case ALLCAP: {
+ ns = pSMgr->ngsuggest(*slst, cw, pHMgr);
+ for (int j=0; j < ns; j++)
+ mkallcap((*slst)[j]);
+ break;
}
- }
+ }
}
- // expand suggestions with dot(s)
- if (abbv) {
- for (int j = 0; j < ns; j++) {
- (*slst)[j] = (char *) realloc((*slst)[j], strlen((*slst)[j]) + 1 + abbv);
- strcat((*slst)[j], word + strlen(word) - abbv);
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ for (int j = 0; j < ns; j++) {
+ if (utf8) reverseword_utf((*slst)[j]); else reverseword((*slst)[j]);
+ }
+ }
+
+ // capitalize and erase capitalized duplications
+ if (capwords) {
+ int l = 0;
+ for (int j=0; j < ns; j++) {
+ mkinitcap((*slst)[j]);
+ (*slst)[l] = (*slst)[j];
+ for (int k=0; k < l; k++) {
+ if (strcmp((*slst)[k], (*slst)[j]) == 0) {
+ free((*slst)[j]);
+ l--;
+ }
}
+ l++;
+ }
+ ns = l;
+ }
+
+ // expand suggestions with dot(s)
+ if (abbv && pAMgr && pAMgr->get_sugswithdots()) {
+ for (int j = 0; j < ns; j++) {
+ (*slst)[j] = (char *) realloc((*slst)[j], strlen((*slst)[j]) + 1 + abbv);
+ strcat((*slst)[j], word + strlen(word) - abbv);
+ }
}
return ns;
@@ -711,16 +754,20 @@ int Hunspell::suggest_auto(char*** slst, const char * word)
}
}
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ for (int j = 0; j < ns; j++) {
+ if (utf8) reverseword_utf((*slst)[j]); else reverseword((*slst)[j]);
+ }
+ }
+
// expand suggestions with dot(s)
- if (abbv) {
- for (int j = 0; j < ns; j++) {
- char * l = (char *) malloc(strlen((*slst)[j]) + 1 + abbv);
- strcpy(l, (*slst)[j]);
- strcat(l, word + strlen(word) - abbv);
- free((*slst)[j]);
- (*slst)[j] = l;
- }
+ if (abbv && pAMgr && pAMgr->get_sugswithdots()) {
+ for (int j = 0; j < ns; j++) {
+ (*slst)[j] = (char *) realloc((*slst)[j], strlen((*slst)[j]) + 1 + abbv);
+ strcat((*slst)[j], word + strlen(word) - abbv);
}
+ }
// replace '-' with ' '
if (forbidden_compound == 2) {
@@ -1032,7 +1079,6 @@ char * Hunspell::morph(const char * word)
}
return mystrdup(result);
}
- // if ((cw[n]=='.') && check(cw + n + 1)) return 1;
switch(captype) {
case NOCAP: {
@@ -1155,12 +1201,18 @@ char * Hunspell::morph(const char * word)
}
}
- if (result && (*result)) return mystrdup(result);
+ if (result && (*result)) {
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ if (utf8) reverseword_utf(result); else reverseword(result);
+ }
+ return mystrdup(result);
+ }
// compound word with dash (HU) I18n
char * dash;
int nresult = 0;
- if ((dash=(char *) strchr(cw,'-'))) {
+ if ((langnum == LANG_hu) && (dash=(char *) strchr(cw,'-'))) {
*dash='\0';
// examine 2 sides of the dash
if (dash[1] == '\0') { // base word ending with dash
@@ -1403,3 +1455,4 @@ int Hunspell::analyze(char ***out, const char *word) {
free(m);
return n;
}
+
diff --git a/src/hunspell/hunspell.hxx b/src/hunspell/hunspell.hxx
index e07cf10..05d5290 100644
--- a/src/hunspell/hunspell.hxx
+++ b/src/hunspell/hunspell.hxx
@@ -33,6 +33,7 @@ class Hunspell
struct unicode_info2 * utfconv;
int langnum;
int utf8;
+ int complexprefixes;
/* XXX not stateless variables for compound handling */
char * prevroot;
@@ -91,6 +92,7 @@ public:
char * get_dic_encoding();
const char * get_wordchars();
struct cs_info * get_csconv();
+ struct unicode_info2 * get_utf_conv();
const char * get_version();
/* experimental functions */
@@ -111,7 +113,7 @@ public:
int suggest_pos_stems(char*** slst, const char * word);
char * get_possible_root();
- /* temporaly, deprecated functions */
+ /* not threadsafe functions for Hunspell command line API */
char * get_prevroot();
int get_prevcompound();
diff --git a/src/hunspell/langnum.hxx b/src/hunspell/langnum.hxx
index ad8e2b7..bad6891 100644
--- a/src/hunspell/langnum.hxx
+++ b/src/hunspell/langnum.hxx
@@ -2,12 +2,12 @@
#define _LANGNUM_HXX_
/*
- language numbers
+ language numbers for language specific codes
see http://l10n.openoffice.org/languages.html
*/
enum {
-LANG_az=100, // custom code
+LANG_az=100, // custom number
LANG_bg=41,
LANG_ca=37,
LANG_cs=42,
@@ -22,7 +22,7 @@ LANG_gl=38,
LANG_hr=78,
LANG_hu=36,
LANG_it=39,
-LANG_la=99, // custom code
+LANG_la=99, // custom number
LANG_nl=31,
LANG_pl=48,
LANG_pt=03,
diff --git a/src/hunspell/makefile.mk b/src/hunspell/makefile.mk
new file mode 100644
index 0000000..1c7ff82
--- /dev/null
+++ b/src/hunspell/makefile.mk
@@ -0,0 +1,112 @@
+#*************************************************************************
+#
+# $RCSfile: makefile.mk,v $
+#
+# $Revision: 1.7 $
+#
+# last change: $Author: vg $ $Date: 2003/06/12 10:38:24 $
+#
+# The Contents of this file are made available subject to the terms of
+# either of the following licenses
+#
+# - GNU Lesser General Public License Version 2.1
+# - Sun Industry Standards Source License Version 1.1
+#
+# Sun Microsystems Inc., October, 2000
+#
+# GNU Lesser General Public License Version 2.1
+# =============================================
+# Copyright 2000 by Sun Microsystems, Inc.
+# 901 San Antonio Road, Palo Alto, CA 94303, USA
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License version 2.1, as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+# MA 02111-1307 USA
+#
+#
+# Sun Industry Standards Source License Version 1.1
+# =================================================
+# The contents of this file are subject to the Sun Industry Standards
+# Source License Version 1.1 (the "License"); You may not use this file
+# except in compliance with the License. You may obtain a copy of the
+# License at http://www.openoffice.org/license.html.
+#
+# Software provided under this License is provided on an "AS IS" basis,
+# WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING,
+# WITHOUT LIMITATION, WARRANTIES THAT THE SOFTWARE IS FREE OF DEFECTS,
+# MERCHANTABLE, FIT FOR A PARTICULAR PURPOSE, OR NON-INFRINGING.
+# See the License for the specific provisions governing your rights and
+# obligations concerning the Software.
+#
+# The Initial Developer of the Original Code is: Sun Microsystems, Inc.
+#
+# Copyright: 2000 by Sun Microsystems, Inc.
+#
+# All Rights Reserved.
+#
+# Contributor(s): _______________________________________
+#
+#
+#
+#*************************************************************************
+
+PRJ = ..
+
+PRJNAME = hunspell
+TARGET = hunspell
+LIBTARGET=NO
+
+#----- Settings ---------------------------------------------------------
+
+.INCLUDE : settings.mk
+
+# --- Files --------------------------------------------------------
+
+# all_target: ALLTAR DICTIONARY
+all_target: ALLTAR
+
+##CXXFLAGS += -I..$/..$/lingutil
+##CFLAGSCXX += -I..$/..$/lingutil
+##CFLAGSCC += -I..$/..$/lingutil
+
+
+SLOFILES= \
+ $(SLO)$/affentry.obj \
+ $(SLO)$/affixmgr.obj \
+ $(SLO)$/dictmgr.obj \
+ $(SLO)$/csutil.obj \
+ $(SLO)$/utf_info.obj \
+ $(SLO)$/hashmgr.obj \
+ $(SLO)$/suggestmgr.obj \
+ $(SLO)$/hunspell.obj
+
+LIB1TARGET= $(SLB)$/lib$(TARGET).lib
+LIB1ARCHIV= $(LB)/lib$(TARGET).a
+LIB1OBJFILES= $(SLOFILES)
+
+# DIC2BIN= \
+# en_US.aff \
+# en_US.dic
+#
+# de_DE.aff \
+# de_DE.dic
+
+
+# DICTIONARY :
+# +$(COPY) $(foreach,i,$(DIC2BIN) $i) $(BIN)
+
+
+# --- Targets ------------------------------------------------------
+
+.INCLUDE : target.mk
+
diff --git a/src/hunspell/suggestmgr.cxx b/src/hunspell/suggestmgr.cxx
index ebe4936..a3a8134 100644
--- a/src/hunspell/suggestmgr.cxx
+++ b/src/hunspell/suggestmgr.cxx
@@ -25,27 +25,35 @@ SuggestMgr::SuggestMgr(const char * tryme, int maxn,
ctry = NULL;
ctry_utf = NULL;
- if (pAMgr && pAMgr->is_utf8()) {
- w_char t[MAXSWL];
- if (tryme) {
- ctryl = u8_u16(t, MAXSWL, tryme);
- ctry_utf = (w_char *) malloc(ctryl * sizeof(w_char));
- memcpy(ctry_utf, t, ctryl * sizeof(w_char));
- }
- } else {
- ctry = mystrdup(tryme);
- if (ctry) ctryl = strlen(ctry);
- }
maxSug = maxn;
nosplitsugs = 0;
- nomapsugs = 0;
+ maxngramsugs = MAXNGRAMSUGS;
+
+ utf8 = 0;
+ utfconv = NULL;
+ complexprefixes = 0;
if (pAMgr) {
char * enc = pAMgr->get_encoding();
csconv = get_current_cs(enc);
free(enc);
nosplitsugs = pAMgr->get_nosplitsugs();
- nomapsugs = pAMgr->get_nomapsugs();
+ if (pAMgr->get_maxngramsugs() >= 0) maxngramsugs = pAMgr->get_maxngramsugs();
+ utf8 = pAMgr->get_utf8();
+ utfconv = pAMgr->get_utf_conv();
+ complexprefixes = pAMgr->get_complexprefixes();
+ }
+
+ if (tryme) {
+ if (utf8) {
+ w_char t[MAXSWL];
+ ctryl = u8_u16(t, MAXSWL, tryme);
+ ctry_utf = (w_char *) malloc(ctryl * sizeof(w_char));
+ memcpy(ctry_utf, t, ctryl * sizeof(w_char));
+ } else {
+ ctry = mystrdup(tryme);
+ ctryl = strlen(ctry);
+ }
}
}
@@ -66,80 +74,81 @@ SuggestMgr::~SuggestMgr()
// generate suggestions for a mispelled word
// pass in address of array of char * pointers
-int SuggestMgr::suggest(char*** slst, const char * word, int nsug)
+int SuggestMgr::suggest(char*** slst, const char * w, int nsug)
{
int nocompoundtwowords = 0;
char ** wlst;
w_char word_utf[MAXSWL];
int wl;
- checknum=0;
-
+ char w2[MAXSWL];
+ const char * word = w;
+
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ strcpy(w2, w);
+ if (utf8) reverseword_utf(w2); else reverseword(w2);
+ word = w2;
+ }
+
if (*slst) {
wlst = *slst;
} else {
- wlst = (char **) calloc(maxSug, sizeof(char *));
+ wlst = (char **) malloc(maxSug * sizeof(char *));
if (wlst == NULL) return -1;
+ for (int i = 0; i < maxSug; i++) wlst[i] = NULL;
}
- if (pAMgr->is_utf8()) {
+ if (utf8) {
wl = u8_u16(word_utf, MAXSWL, word);
}
- for (cpdsuggest=0; (cpdsuggest<2) && (nocompoundtwowords==0); cpdsuggest++) {
-
- // perhaps we made chose the wrong char from a related set
- if ((!nomapsugs) && (nsug < maxSug) && (nsug > -1))
- nsug = mapchars(wlst, word, nsug);
+ for (int cpdsuggest=0; (cpdsuggest<2) && (nocompoundtwowords==0); cpdsuggest++) {
// perhaps we made a typical fault of spelling
if ((nsug < maxSug) && (nsug > -1))
- nsug = replchars(wlst, word, nsug);
+ nsug = replchars(wlst, word, nsug, cpdsuggest);
- // perhaps we made a special pattern mistake (HU)
+ // perhaps we made chose the wrong char from a related set
if ((nsug < maxSug) && (nsug > -1))
-/// nsug = doubledsyllable(wlst, word, nsug);
+ nsug = mapchars(wlst, word, nsug, cpdsuggest);
- // perhaps we wrote without accents (typical in e-mails) (HU)
- if ((cpdsuggest == 0) && (nsug < maxSug) && (nsug > -1))
-/// nsug = forgotaccent(wlst, word, nsug);
+ // perhaps we made a special pattern mistake
+ // if ((nsug < maxSug) && (nsug > -1))
+ // nsug = doubledsyllable(wlst, word, nsug);
// did we forget to add a char
-// if ((cpdsuggest == 0) && (nsug < maxSug) && (nsug > -1))
if ((nsug < maxSug) && (nsug > -1)) {
- nsug = (pAMgr->is_utf8()) ? forgotchar_utf(wlst, word_utf, wl, nsug) :
- forgotchar(wlst, word, nsug);
+ nsug = (utf8) ? forgotchar_utf(wlst, word_utf, wl, nsug, cpdsuggest) :
+ forgotchar(wlst, word, nsug, cpdsuggest);
}
// did we swap the order of chars by mistake
if ((nsug < maxSug) && (nsug > -1)) {
- nsug = (pAMgr->is_utf8()) ? swapchar_utf(wlst, word_utf, wl, nsug) :
- swapchar(wlst, word, nsug);
+ nsug = (utf8) ? swapchar_utf(wlst, word_utf, wl, nsug, cpdsuggest) :
+ swapchar(wlst, word, nsug, cpdsuggest);
}
// did we add a char that should not be there
if ((nsug < maxSug) && (nsug > -1)) {
- nsug = (pAMgr->is_utf8()) ? extrachar_utf(wlst, word_utf, wl, nsug) :
- extrachar(wlst, word, nsug);
+ nsug = (utf8) ? extrachar_utf(wlst, word_utf, wl, nsug, cpdsuggest) :
+ extrachar(wlst, word, nsug, cpdsuggest);
}
// did we just hit the wrong key in place of a good char
if ((nsug < maxSug) && (nsug > -1)) {
- nsug = (pAMgr->is_utf8()) ? badchar_utf(wlst, word_utf, wl, nsug) :
- badchar(wlst, word, nsug);
+ nsug = (utf8) ? badchar_utf(wlst, word_utf, wl, nsug, cpdsuggest) :
+ badchar(wlst, word, nsug, cpdsuggest);
}
if ((cpdsuggest==0) && (nsug>0)) nocompoundtwowords=1;
// perhaps we forgot to hit space and two words ran together
if ((!nosplitsugs) && (nsug < maxSug) && (nsug > -1)) {
- int rchecknum = checknum;
- checknum = 0;
- nsug = twowords(wlst, word, nsug);
- checknum = rchecknum;
+ nsug = twowords(wlst, word, nsug, cpdsuggest);
}
-
+
} // repeating ``for'' statement compounding support
if (nsug < 0) {
@@ -156,63 +165,44 @@ int SuggestMgr::suggest(char*** slst, const char * word, int nsug)
// generate suggestions for a word with typical mistake
// pass in address of array of char * pointers
-int SuggestMgr::suggest_auto(char*** slst, const char * word, int nsug)
+int SuggestMgr::suggest_auto(char*** slst, const char * w, int nsug)
{
int nocompoundtwowords = 0;
- char ** wlst;
- checknum=0;
+ char ** wlst;
+
+ char w2[MAXSWL];
+ const char * word = w;
+
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ strcpy(w2, w);
+ if (utf8) reverseword_utf(w2); else reverseword(w2);
+ word = w2;
+ }
if (*slst) {
wlst = *slst;
} else {
- wlst = (char **) calloc(maxSug, sizeof(char *));
+ wlst = (char **) malloc(maxSug * sizeof(char *));
if (wlst == NULL) return -1;
}
- for (cpdsuggest=0; (cpdsuggest<2) && (nocompoundtwowords==0); cpdsuggest++) {
+ for (int cpdsuggest=0; (cpdsuggest<2) && (nocompoundtwowords==0); cpdsuggest++) {
// perhaps we made a typical fault of spelling
if ((nsug < maxSug) && (nsug > -1))
- nsug = replchars(wlst, word, nsug);
-
- // perhaps we made a special pattern mistake (HU)
- if ((nsug < maxSug) && (nsug > -1))
-/// nsug = doubledsyllable(wlst, word, nsug);
-
-/*
- // perhaps we wrote without accents (typical in e-mails) (HU)
- if ((nsug < maxSug) && (nsug > -1))
- nsug = forgotaccent(wlst, word, nsug);
-
- // did we forget to add a char
- if ((nsug < maxSug) && (nsug > -1))
- nsug = forgotchar(wlst, word, nsug);
-
- // did we swap the order of chars by mistake
- if ((nsug < maxSug) && (nsug > -1))
- nsug = swapchar(wlst, word, nsug);
-
- // did we add a char that should not be there
- if ((nsug < maxSug) && (nsug > -1))
- nsug = extrachar(wlst, word, nsug);
-
- // did we just hit the wrong key in place of a good char
- if ((nsug < maxSug) && (nsug > -1))
- nsug = badchar(wlst, word, nsug);
+ nsug = replchars(wlst, word, nsug, cpdsuggest);
-*/
+ // perhaps we made chose the wrong char from a related set
+ if ((nsug < maxSug) && (nsug > -1) && (cpdsuggest == 0))
+ nsug = mapchars(wlst, word, nsug, cpdsuggest);
if ((cpdsuggest==0) && (nsug>0)) nocompoundtwowords=1;
- // diszhal, disz hal,
// perhaps we forgot to hit space and two words ran together
- if (//( (cpdsuggest==0) || (nsug==0)) &&
- (nsug < maxSug) && (nsug > -1) && check_forbidden(word, strlen(word))) {
- int rchecknum = checknum;
- checknum = 0;
- nsug = twowords(wlst, word, nsug);
- checknum = rchecknum;
+ if ((nsug < maxSug) && (nsug > -1) && check_forbidden(word, strlen(word))) {
+ nsug = twowords(wlst, word, nsug, cpdsuggest);
}
} // repeating ``for'' statement compounding support
@@ -230,27 +220,37 @@ int SuggestMgr::suggest_auto(char*** slst, const char * word, int nsug)
// suggestions for when chose the wrong char out of a related set
-int SuggestMgr::mapchars(char** wlst, const char * word, int ns)
+int SuggestMgr::mapchars(char** wlst, const char * word, int ns, int cpdsuggest)
{
+ time_t timelimit;
+ int timer;
+
int wl = strlen(word);
if (wl < 2 || ! pAMgr) return ns;
int nummap = pAMgr->get_nummap();
struct mapentry* maptable = pAMgr->get_maptable();
if (maptable==NULL) return ns;
- ns = map_related(word, 0, wlst, ns, maptable, nummap);
+
+ timelimit = time(NULL);
+ timer = MINTIMER;
+ if (utf8) {
+ w_char w[MAXSWL];
+ int len = u8_u16(w, MAXSWL, word);
+ ns = map_related_utf(w, len, 0, wlst, ns, maptable, nummap, &timer, &timelimit);
+ } else ns = map_related(word, 0, wlst, ns, maptable, nummap, &timer, &timelimit);
return ns;
}
-
-int SuggestMgr::map_related(const char * word, int i, char** wlst, int ns, const mapentry* maptable, int nummap)
+int SuggestMgr::map_related(const char * word, int i, char** wlst, int ns,
+ const mapentry* maptable, int nummap, int * timer, time_t * timelimit)
{
char c = *(word + i);
if (c == 0) {
- int cwrd = 1;
+ int cwrd = 1;
for (int m=0; m < ns; m++)
if (strcmp(word,wlst[m]) == 0) cwrd = 0;
- if ((cwrd) && check(word,strlen(word))) {
+ if ((cwrd) && check(word,strlen(word), 1, timer, timelimit)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(word);
if (wlst[ns] == NULL) return -1;
@@ -263,27 +263,65 @@ int SuggestMgr::map_related(const char * word, int i, char** wlst, int ns, const
for (int j = 0; j < nummap; j++) {
if (strchr(maptable[j].set,c) != 0) {
in_map = 1;
- char * newword = strdup(word);
+ char * newword = mystrdup(word);
for (int k = 0; k < maptable[j].len; k++) {
*(newword + i) = *(maptable[j].set + k);
- ns = map_related(newword, (i+1), wlst, ns, maptable, nummap);
+ ns = map_related(newword, (i+1), wlst, ns, maptable, nummap, timer, timelimit);
+ if (!(*timelimit)) return ns;
}
free(newword);
}
}
if (!in_map) {
i++;
- ns = map_related(word, i, wlst, ns, maptable, nummap);
+ ns = map_related(word, i, wlst, ns, maptable, nummap, timer, timelimit);
}
return ns;
}
+int SuggestMgr::map_related_utf(w_char * word, int len, int i, char** wlst, int ns,
+ const mapentry* maptable, int nummap, int * timer, time_t * timelimit)
+{
+ if (i == len) {
+ int cwrd = 1;
+ char s[MAXSWL];
+ u16_u8(s, MAXSWL, word, len);
+ for (int m=0; m < ns; m++)
+ if (strcmp(s,wlst[m]) == 0) cwrd = 0;
+ if ((cwrd) && check(s,strlen(s), 1, timer, timelimit)) {
+ if (ns < maxSug) {
+ wlst[ns] = mystrdup(s);
+ if (wlst[ns] == NULL) return -1;
+ ns++;
+ }
+ }
+ return ns;
+ }
+ int in_map = 0;
+ unsigned short c = *((unsigned short *) word + i);
+ for (int j = 0; j < nummap; j++) {
+ if (flag_bsearch((unsigned short *) maptable[j].set_utf16, c, maptable[j].len)) {
+ in_map = 1;
+ for (int k = 0; k < maptable[j].len; k++) {
+ *(word + i) = *(maptable[j].set_utf16 + k);
+ ns = map_related_utf(word, len, i + 1, wlst, ns, maptable, nummap, timer, timelimit);
+ if (!(*timelimit)) return ns;
+ }
+ *((unsigned short *) word + i) = c;
+ }
+ }
+ if (!in_map) {
+ i++;
+ ns = map_related_utf(word, len, i, wlst, ns, maptable, nummap, timer, timelimit);
+ }
+ return ns;
+}
// suggestions for a typical fault of spelling, that
// differs with more, than 1 letter from the right form.
-int SuggestMgr::replchars(char** wlst, const char * word, int ns)
+int SuggestMgr::replchars(char** wlst, const char * word, int ns, int cpdsuggest)
{
char candidate[MAXSWL];
const char * r;
@@ -310,7 +348,7 @@ int SuggestMgr::replchars(char** wlst, const char * word, int ns)
cwrd = 1;
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate,strlen(candidate))) {
+ if ((cwrd) && check(candidate,strlen(candidate), cpdsuggest, NULL, NULL)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) {
@@ -326,11 +364,9 @@ int SuggestMgr::replchars(char** wlst, const char * word, int ns)
return ns;
}
-// perhaps we made a special pattern mistake (HU)
-// for example: iktat�s -> iktatat�s (doubled `ta')
-// The doubled syllables have a special neural background
-// for example: i k t a t [reactivated `ta' pattern, very similar to `t�'] a t � s
-int SuggestMgr::doubledsyllable(char** wlst, const char * word, int ns)
+// perhaps we made a special pattern mistake
+// for example: vacation -> vacacation (doubled `ac')
+int SuggestMgr::doubledsyllable(char** wlst, const char * word, int ns, int cpdsuggest)
{
char candidate[MAXSWL];
int state=0;
@@ -348,7 +384,7 @@ int SuggestMgr::doubledsyllable(char** wlst, const char * word, int ns)
cwrd = 1;
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate,strlen(candidate))) {
+ if ((cwrd) && check(candidate,strlen(candidate), cpdsuggest, NULL, NULL)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) {
@@ -367,80 +403,13 @@ int SuggestMgr::doubledsyllable(char** wlst, const char * word, int ns)
return ns;
}
-int SuggestMgr::permuteaccent(char ** wlst, char * word, int ns, int i,
- int wl, replentry * rep, int depth) {
-
- int cwrd;
-// int accentlen = strlen(rep->pattern);
-
- while ((word[i]!='\0') &&
- (! strchr(rep->replacement, word[i]))) i++;
-
- if ((word[i]=='\0') || (depth==MAXACCENT)) {
- cwrd = 1;
-
- for (int k=0; k < ns; k++)
- if (strcmp(word,wlst[k]) == 0) cwrd = 0;
-
- if ((cwrd) && check(word,wl)) {
- if (ns < maxSug) {
- wlst[ns] = mystrdup(word);
- if (wlst[ns] == NULL) return -1;
- ns++;
- } else return ns;
- }
- } else {
- if ((ns=permuteaccent(wlst, word, ns, i+1, wl, rep, depth+1))==-1) return -1;
- char tmpc=word[i];
- char * accentpos = rep->replacement - 1;
- for (;;) {
-// accentpos = (char *) TESTAFF(accentpos+1, tmpc, accentlen - (accentpos - rep->replacement));
- accentpos = strchr(accentpos+1, tmpc);
- if (accentpos==NULL) {
- word[i]=tmpc;
- return ns;
- }
- word[i]=(rep->pattern)[accentpos-(rep->replacement)];
- if ((ns=permuteaccent(wlst, word, ns, i+1, wl, rep, depth+1))==-1) return -1;
- }
- }
- return ns;
-}
-
-// forgot accents
-int SuggestMgr::forgotaccent(char ** wlst, const char * word, int ns)
-{
- char candidate[MAXSWL];
- replentry * rep;
-// int accpos[MAXACCENT];
-
- int wl = strlen(word);
- strcpy (candidate, word);
-
- if ((pAMgr==NULL) || (rep=pAMgr->get_accent())==NULL) return ns;
-
-/*
- int accentlen = strlen(rep->pattern);
- int j=0;
- for (int i=0; i<wl && n<MAXACCENT ; i++) {
- if (TESTAFF(replentry->replacement, candidate[i], accentlen) {
- accpos[j]=i;
- j++;
- }
- }
-*/
-
- if ((ns=permuteaccent(wlst, candidate, ns, 0, wl, rep, 0))==-1) return -1;
-
- return ns;
-}
-
-
// error is wrong char in place of correct one
-int SuggestMgr::badchar(char ** wlst, const char * word, int ns)
+int SuggestMgr::badchar(char ** wlst, const char * word, int ns, int cpdsuggest)
{
char tmpc;
char candidate[MAXSWL];
+ time_t timelimit = time(NULL);
+ int timer = MINTIMER;
int wl = strlen(word);
int cwrd;
@@ -456,13 +425,14 @@ int SuggestMgr::badchar(char ** wlst, const char * word, int ns)
cwrd = 1;
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate,wl)) {
+ if ((cwrd) && check(candidate,wl, cpdsuggest, &timer, &timelimit)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
ns++;
} else return ns;
}
+ if (!timelimit) return ns;
candidate[i] = tmpc;
}
}
@@ -470,13 +440,14 @@ int SuggestMgr::badchar(char ** wlst, const char * word, int ns)
}
// error is wrong char in place of correct one
-int SuggestMgr::badchar_utf(char ** wlst, const w_char * word, int wl, int ns)
+int SuggestMgr::badchar_utf(char ** wlst, const w_char * word, int wl, int ns, int cpdsuggest)
{
w_char tmpc;
w_char candidate_utf[MAXSWL];
char candidate[MAXSWL];
-
int cwrd;
+ time_t timelimit = time(NULL);
+ int timer = MINTIMER;
memcpy(candidate_utf, word, wl * sizeof(w_char));
@@ -491,13 +462,14 @@ int SuggestMgr::badchar_utf(char ** wlst, const w_char * word, int wl, int ns)
u16_u8(candidate, MAXSWL, candidate_utf, wl);
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate, strlen(candidate))) {
+ if ((cwrd) && check(candidate, strlen(candidate), cpdsuggest, &timer, &timelimit)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
ns++;
} else return ns;
}
+ if (!timelimit) return ns;
candidate_utf[i] = tmpc;
}
}
@@ -505,7 +477,7 @@ int SuggestMgr::badchar_utf(char ** wlst, const w_char * word, int wl, int ns)
}
// error is word has an extra letter it does not need
-int SuggestMgr::extrachar_utf(char** wlst, const w_char * word, int wl, int ns)
+int SuggestMgr::extrachar_utf(char** wlst, const w_char * word, int wl, int ns, int cpdsuggest)
{
char candidate[MAXSWL];
w_char candidate_utf[MAXSWL];
@@ -523,7 +495,7 @@ int SuggestMgr::extrachar_utf(char** wlst, const w_char * word, int wl, int ns)
u16_u8(candidate, MAXSWL, candidate_utf, wl - 1);
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate, strlen(candidate))) {
+ if ((cwrd) && check(candidate, strlen(candidate), cpdsuggest, NULL, NULL)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
@@ -536,7 +508,7 @@ int SuggestMgr::extrachar_utf(char** wlst, const w_char * word, int wl, int ns)
}
// error is word has an extra letter it does not need
-int SuggestMgr::extrachar(char** wlst, const char * word, int ns)
+int SuggestMgr::extrachar(char** wlst, const char * word, int ns, int cpdsuggest)
{
char candidate[MAXSWL];
const char * p;
@@ -552,7 +524,7 @@ int SuggestMgr::extrachar(char** wlst, const char * word, int ns)
cwrd = 1;
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate,wl-1)) {
+ if ((cwrd) && check(candidate,wl-1, cpdsuggest, NULL, NULL)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
@@ -566,12 +538,14 @@ int SuggestMgr::extrachar(char** wlst, const char * word, int ns)
// error is missing a letter it needs
-int SuggestMgr::forgotchar(char ** wlst, const char * word, int ns)
+int SuggestMgr::forgotchar(char ** wlst, const char * word, int ns, int cpdsuggest)
{
char candidate[MAXSWL];
const char * p;
char * q;
int cwrd;
+ time_t timelimit = time(NULL);
+ int timer = MINTIMER;
int wl = strlen(word);
@@ -583,13 +557,14 @@ int SuggestMgr::forgotchar(char ** wlst, const char * word, int ns)
cwrd = 1;
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate,wl+1)) {
+ if ((cwrd) && check(candidate, wl+1, cpdsuggest, &timer, &timelimit)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
ns++;
} else return ns;
}
+ if (!timelimit) return ns;
}
*q++ = *p++;
}
@@ -600,7 +575,7 @@ int SuggestMgr::forgotchar(char ** wlst, const char * word, int ns)
cwrd = 1;
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate,wl+1)) {
+ if ((cwrd) && check(candidate,wl+1, cpdsuggest, NULL, NULL)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
@@ -612,13 +587,15 @@ int SuggestMgr::forgotchar(char ** wlst, const char * word, int ns)
}
// error is missing a letter it needs
-int SuggestMgr::forgotchar_utf(char ** wlst, const w_char * word, int wl, int ns)
+int SuggestMgr::forgotchar_utf(char ** wlst, const w_char * word, int wl, int ns, int cpdsuggest)
{
w_char candidate_utf[MAXSWL];
char candidate[MAXSWL];
const w_char * p;
w_char * q;
int cwrd;
+ time_t timelimit = time(NULL);
+ int timer = MINTIMER;
// try inserting a tryme character before every letter
memcpy (candidate_utf + 1, word, wl * sizeof(w_char));
@@ -628,15 +605,16 @@ int SuggestMgr::forgotchar_utf(char ** wlst, const w_char * word, int wl, int ns
cwrd = 1;
u16_u8(candidate, MAXSWL, candidate_utf, wl + 1);
for (int k=0; k < ns; k++)
- if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate, strlen(candidate))) {
- if (ns < maxSug) {
+ if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
+ if ((cwrd) && check(candidate, strlen(candidate), cpdsuggest, &timer, &timelimit)) {
+ if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
ns++;
} else return ns;
}
- }
+ if (!timelimit) return ns;
+ }
*q++ = *p++;
}
@@ -647,7 +625,7 @@ int SuggestMgr::forgotchar_utf(char ** wlst, const w_char * word, int wl, int ns
u16_u8(candidate, MAXSWL, candidate_utf, wl + 1);
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate, strlen(candidate))) {
+ if ((cwrd) && check(candidate, strlen(candidate), cpdsuggest, NULL, NULL)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
@@ -660,13 +638,12 @@ int SuggestMgr::forgotchar_utf(char ** wlst, const w_char * word, int wl, int ns
/* error is should have been two words */
-int SuggestMgr::twowords(char ** wlst, const char * word, int ns)
+int SuggestMgr::twowords(char ** wlst, const char * word, int ns, int cpdsuggest)
{
char candidate[MAXSWL];
char * p;
int c1, c2, cwrd;
int forbidden = 0;
- int utf8 = pAMgr->is_utf8();
int wl=strlen(word);
if (wl < 4) return ns;
@@ -686,8 +663,8 @@ int SuggestMgr::twowords(char ** wlst, const char * word, int ns)
p[-1] = *p;
}
*p = '\0';
- if ((c1=check(candidate,strlen(candidate)))) {
- if ((c2=check((p+1),strlen(p+1)))) {
+ if ((c1=check(candidate,strlen(candidate), cpdsuggest, NULL, NULL))) {
+ if ((c2=check((p+1),strlen(p+1), cpdsuggest, NULL, NULL))) {
*p = ' ';
// spec. Hungarian code (need a better compound word support)
@@ -715,7 +692,7 @@ int SuggestMgr::twowords(char ** wlst, const char * word, int ns)
// error is adjacent letter were swapped
-int SuggestMgr::swapchar(char ** wlst, const char * word, int ns)
+int SuggestMgr::swapchar(char ** wlst, const char * word, int ns, int cpdsuggest)
{
char candidate[MAXSWL];
char * p;
@@ -733,7 +710,7 @@ int SuggestMgr::swapchar(char ** wlst, const char * word, int ns)
cwrd = 1;
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate,wl)) {
+ if ((cwrd) && check(candidate,wl, cpdsuggest, NULL, NULL)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
@@ -748,7 +725,7 @@ int SuggestMgr::swapchar(char ** wlst, const char * word, int ns)
}
// error is adjacent letter were swapped
-int SuggestMgr::swapchar_utf(char ** wlst, const w_char * word, int wl, int ns)
+int SuggestMgr::swapchar_utf(char ** wlst, const w_char * word, int wl, int ns, int cpdsuggest)
{
w_char candidate_utf[MAXSWL];
char candidate[MAXSWL];
@@ -766,7 +743,7 @@ int SuggestMgr::swapchar_utf(char ** wlst, const w_char * word, int wl, int ns)
u16_u8(candidate, MAXSWL, candidate_utf, wl);
for (int k=0; k < ns; k++)
if (strcmp(candidate,wlst[k]) == 0) cwrd = 0;
- if ((cwrd) && check(candidate, strlen(candidate))) {
+ if ((cwrd) && check(candidate, strlen(candidate), cpdsuggest, NULL, NULL)) {
if (ns < maxSug) {
wlst[ns] = mystrdup(candidate);
if (wlst[ns] == NULL) return -1;
@@ -781,7 +758,7 @@ int SuggestMgr::swapchar_utf(char ** wlst, const w_char * word, int wl, int ns)
}
// generate a set of suggestions for very poorly spelled words
-int SuggestMgr::ngsuggest(char** wlst, char * word, HashMgr* pHMgr)
+int SuggestMgr::ngsuggest(char** wlst, char * w, HashMgr* pHMgr)
{
int i, j;
@@ -801,14 +778,29 @@ int SuggestMgr::ngsuggest(char** wlst, char * word, HashMgr* pHMgr)
}
lp = MAX_ROOTS - 1;
- int n = strlen(word);
+ char w2[MAXSWL];
+ char * word = w;
+
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ strcpy(w2, w);
+ if (utf8) reverseword_utf(w2); else reverseword(w2);
+ word = w2;
+ }
+
+ char mw[MAXSWL];
+ w_char u8[MAXSWL];
+ int nc = strlen(word);
+ int n = (utf8) ? u8_u16(u8, MAXSWL, word) : nc;
struct hentry* hp = NULL;
int col = -1;
while ((hp = pHMgr->walk_hashtable(col, hp))) {
+ // check forbidden words
+ if ((hp->astr) && (pAMgr) && TESTAFF(hp->astr, pAMgr->get_forbiddenword(), hp->alen)) continue;
sc = ngram(3, word, hp->word, NGRAM_LONGER_WORSE);
if (sc > scores[lp]) {
- scores[lp] = sc;
+ scores[lp] = sc;
roots[lp] = hp;
int lval = sc;
for (j=0; j < MAX_ROOTS; j++)
@@ -823,14 +815,18 @@ int SuggestMgr::ngsuggest(char** wlst, char * word, HashMgr* pHMgr)
// mangle original word three differnt ways
// and score them to generate a minimum acceptable score
int thresh = 0;
- char * mw = NULL;
+// int n = (utf8) ? u8_u16(u8, MAXSWL, word) : strlen(word);
for (int sp = 1; sp < 4; sp++) {
- mw = strdup(word);
- for (int k=sp; k < n; k+=4) *(mw + k) = '*';
- thresh = thresh + ngram(n, word, mw, NGRAM_ANY_MISMATCH);
- free(mw);
+ if (utf8) {
+ for (int k=sp; k < n; k+=4) *((unsigned short *) u8 + k) = '*';
+ u16_u8(mw, MAXSWL, u8, n);
+ thresh = thresh + ngram(n, word, mw, NGRAM_ANY_MISMATCH);
+ } else {
+ strcpy(mw, word);
+ for (int k=sp; k < n; k+=4) *(mw + k) = '*';
+ thresh = thresh + ngram(n, word, mw, NGRAM_ANY_MISMATCH);
+ }
}
- mw = NULL;
thresh = thresh / 3;
thresh--;
@@ -855,10 +851,11 @@ int SuggestMgr::ngsuggest(char** wlst, char * word, HashMgr* pHMgr)
if (roots[i]) {
struct hentry * rp = roots[i];
int nw = pAMgr->expand_rootword(glst, MAX_WORDS, rp->word, rp->wlen,
- rp->astr, rp->alen);
- for (int k = 0; k < nw; k++) {
+ rp->astr, rp->alen, word, nc);
+
+ for (int k = 0; k < nw ; k++) {
sc = ngram(n, word, glst[k].word, NGRAM_ANY_MISMATCH);
- if (sc > thresh) {
+ if ((sc > thresh)) {
if (sc > gscore[lp]) {
if (guess[lp]) free (guess[lp]);
gscore[lp] = sc;
@@ -869,59 +866,85 @@ int SuggestMgr::ngsuggest(char** wlst, char * word, HashMgr* pHMgr)
lp = j;
lval = gscore[j];
}
- } else {
- free (glst[k].word);
- }
- }
+ } else free (glst[k].word);
+ } else free(glst[k].word);
}
}
}
if (glst) free(glst);
// now we are done generating guesses
- // sort in order of decreasing score and copy over
+ // sort in order of decreasing score
bubblesort(&guess[0], &gscore[0], MAX_GUESS);
+
+ // weight suggestions with a similarity index, based on
+ // the longest common subsequent algorithm and resort
+
+ for (i=0; i < MAX_GUESS; i++) {
+ if (guess[i]) {
+ int lcs = lcslen(word, guess[i]);
+ gscore[i] +=
+ // length of longest common subsequent minus lenght difference
+ 2 * lcs - abs((int) (n - mystrlen(guess[i]))) +
+ // weight equal first letter
+ equalfirstletter(word, guess[i]) +
+ // weight equal character positions
+ ((lcs == commoncharacterpositions(word, guess[i])) ? 1: 0);
+ }
+ }
+
+ bubblesort(&guess[0], &gscore[0], MAX_GUESS);
+
+ // copy over
+
int ns = 0;
for (i=0; i < MAX_GUESS; i++) {
if (guess[i]) {
- int unique = 1;
- for (j=i+1; j < MAX_GUESS; j++)
- if (guess[j])
- if (!strcmp(guess[i], guess[j])) unique = 0;
- if (unique) {
- wlst[ns++] = guess[i];
- } else {
- free(guess[i]);
- }
+ if ((ns < maxngramsugs) && (ns < maxSug)) {
+ int unique = 1;
+ for (j=0; j < ns; j++)
+ // don't suggest previous suggestions or a previous suggestion with prefixes or affixes
+ if (strstr(guess[i], wlst[j]) ||
+ // check forbidden words
+ !check(guess[i], strlen(guess[i]), 0, NULL, NULL)) unique = 0;
+ if (unique) wlst[ns++] = guess[i]; else free(guess[i]);
+ } else free(guess[i]);
}
}
+
return ns;
}
-
-
-
// see if a candidate suggestion is spelled correctly
// needs to check both root words and words with affixes
-// MySpell-HU modifications:
+// obsolote MySpell-HU modifications:
// return value 2 and 3 marks compounding with hyphen (-)
// `3' marks roots without suffix
-int SuggestMgr::check(const char * word, int len)
+int SuggestMgr::check(const char * word, int len, int cpdsuggest, int * timer, time_t * timelimit)
{
struct hentry * rv=NULL;
int nosuffix = 0;
- checknum++;
- if ((cpdsuggest==1) && (checknum>6500)) return 0;
+
+ // check time limit
+ if (timer) {
+ (*timer)--;
+ if (!(*timer) && timelimit) {
+ if (time(NULL) > *timelimit) {
+ *timelimit = 0;
+ return 0;
+ }
+ *timer = MAXPLUSTIMER;
+ }
+ }
if (pAMgr) {
rv = pAMgr->lookup(word);
if (rv) {
- if (rv->astr)
if ((rv->astr) && (TESTAFF(rv->astr,pAMgr->get_forbiddenword(),rv->alen))) return 0;
if (rv->astr && TESTAFF(rv->astr,pAMgr->get_pseudoroot(),rv->alen)) rv = NULL;
} else rv = pAMgr->prefix_check(word,len,1); // only prefix, and prefix + suffix XXX
@@ -929,7 +952,6 @@ int SuggestMgr::check(const char * word, int len)
if (rv) {
nosuffix=1;
} else {
- // rv = pAMgr->affix_check(word,len); // prefix+suffix, suffix
rv = pAMgr->suffix_check(word, len, 0, NULL, NULL, 0, NULL); // only suffix
}
@@ -938,25 +960,20 @@ int SuggestMgr::check(const char * word, int len)
if (!rv) rv = pAMgr->prefix_check_twosfx(word, len, 1, FLAG_NULL);
}
- // check forbidden words (HU)
+ // check forbidden words
if ((rv) && (rv->astr) && TESTAFF(rv->astr,pAMgr->get_forbiddenword(),rv->alen)
&& (! TESTAFF(rv->astr,pAMgr->get_onlyroot(),rv->alen))
-/// && (strcmp(word,rv->word)==0) // forbidden root only
) return 0;
if (cpdsuggest==1) {
- if ((rv == NULL) && (pAMgr->get_compoundflag() || pAMgr->get_compoundbegin())) {
+ if ((!rv) && (pAMgr->get_compoundflag() || pAMgr->get_compoundbegin())) {
rv = pAMgr->compound_check(word,len,0,0,0,0,NULL,NULL);
- if (rv) return 3;
+ if (rv) return 3; // XXX obsolote
}
}
}
- if (rv) {
-// if (cpd) {
-// compoundroot = pAMgr->get_compoundroot();
-// if ((compoundroot) && TESTAFF(rv->astr, compoundroot[0], rv->alen)) return 2;
-// }
+ if (rv) { // XXX obsolote
if ((pAMgr->get_compoundflag()) &&
TESTAFF(rv->astr, pAMgr->get_compoundflag(), rv->alen)) {
return 2 + nosuffix;
@@ -969,25 +986,35 @@ int SuggestMgr::check(const char * word, int len)
int SuggestMgr::check_forbidden(const char * word, int len)
{
struct hentry * rv = NULL;
-
+
if (pAMgr) {
rv = pAMgr->lookup(word);
if (rv) if (rv->astr && TESTAFF(rv->astr,pAMgr->get_pseudoroot(),rv->alen)) rv = NULL;
if (!(pAMgr->prefix_check(word,len,1)))
rv = pAMgr->suffix_check(word,len, 0, NULL, NULL, 0, NULL); // prefix+suffix, suffix
- // check forbidden words (HU)
+ // check forbidden words
if ((rv) && (rv->astr) && TESTAFF(rv->astr,pAMgr->get_forbiddenword(),rv->alen)) return 1;
}
return 0;
}
-// suggest stems
-int SuggestMgr::suggest_stems(char*** slst, const char * word, int nsug)
+// suggest stems, XXX experimental code
+int SuggestMgr::suggest_stems(char*** slst, const char * w, int nsug)
{
char buf[MAXSWL];
char ** wlst;
int prevnsug = nsug;
+ char w2[MAXSWL];
+ const char * word = w;
+
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ strcpy(w2, w);
+ if (utf8) reverseword_utf(w2); else reverseword(w2);
+ word = w2;
+ }
+
if (*slst) {
wlst = *slst;
} else {
@@ -1080,7 +1107,7 @@ int SuggestMgr::fixstems(char ** wlst, const char * word, int ns)
strcpy(prefix, pAMgr->get_prefix());
}
- // ha a prefix leg, akkor ezt nem t�ntetj�k fel a t�vekhez
+ // XXX obsolote, will be a general solution for stemming
if ((prefix) && (strncmp(prefix, "leg", 3)==0)) prefix[0] = '\0'; // (HU)
}
@@ -1095,8 +1122,7 @@ int SuggestMgr::fixstems(char ** wlst, const char * word, int ns)
if ((ns < maxSug) && (dicstem < 2)) {
strcpy(buf, prefix);
if ((dicstem > 0) && pAMgr->get_derived()) {
- // XXX (HU) Hung. prefix with cutting, Mak� -> mak�i
- // need a condition for HU lang.
+ // XXX obsolote
if (strlen(prefix) == 1) {
strcat(buf, (pAMgr->get_derived()) + 1);
} else {
@@ -1151,8 +1177,7 @@ int SuggestMgr::fixstems(char ** wlst, const char * word, int ns)
}
}
while (rv) {
-// if (rv->astr && (p = (char *) strchr(rv->astr, 'g'))) { // (HU) XXX 'g' bedr�tozva
- if (0) { // (HU) XXX 'g' bedr�tozva
+ if (0) { // obsolote
if ((p[1] > '0') && (p[1] <= '9')) {
if ((ns < maxSug) && (dicstem != 2)) {
int split = p[1] - '0';
@@ -1160,7 +1185,7 @@ while (rv) {
strcpy(fix, rv->word);
- // ikes ig�k ellen�rz�se
+ // checking verbs ending with `ik'
fix[rv->wlen - split] = 'i';
fix[rv->wlen - split + 1] = 'k';
@@ -1177,8 +1202,7 @@ while (rv) {
}
-// if (strchr(rv2->astr, '0') == NULL) { // (HU) XXX '0' bedr�tozva
- if (0) { // (HU) XXX '0' bedr�tozva
+ if (0) {
strcpy(buf, prefix);
strcat(buf, fix);
wlst[ns] = mystrdup(buf);
@@ -1196,7 +1220,6 @@ while (rv) {
rv2 = pAMgr->lookup(fix);
if ((rv2) && (rv2->astr) && (ns < maxSug))
if ((rv2) && (rv2->astr) && (ns < maxSug))
-// if (strchr(rv2->astr, '|') == NULL) {
if (0) {
char buf2[MAXSWL];
@@ -1204,9 +1227,7 @@ while (rv) {
if (*(rv2->astr) == '-') {
strcat(buf2, "");
-// strcat(buf2, (rv2->astr) + 1);
} else {
-// strcat(buf2, rv2->astr);
strcat(buf2, "");
}
@@ -1231,7 +1252,6 @@ while (rv) {
}
// many stems
} else {
-// char * str = mystrdup(rv2->astr);
char * str = mystrdup("");
char * pos = str;
char * pos2;
@@ -1272,14 +1292,25 @@ return ns;
}
// suggest possible stems
-int SuggestMgr::suggest_pos_stems(char*** slst, const char * word, int nsug)
+int SuggestMgr::suggest_pos_stems(char*** slst, const char * w, int nsug)
{
char ** wlst;
struct hentry * rv = NULL;
+ char w2[MAXSWL];
+ const char * word = w;
+
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ strcpy(w2, w);
+ if (utf8) reverseword_utf(w2); else reverseword(w2);
+ word = w2;
+ }
+
int wl = strlen(word);
+
if (*slst) {
wlst = *slst;
} else {
@@ -1301,7 +1332,7 @@ int SuggestMgr::suggest_pos_stems(char*** slst, const char * word, int nsug)
}
-char * SuggestMgr::suggest_morph(const char * word)
+char * SuggestMgr::suggest_morph(const char * w)
{
char result[MAXLNLEN];
char * r = (char *) result;
@@ -1313,6 +1344,16 @@ char * SuggestMgr::suggest_morph(const char * word)
if (! pAMgr) return NULL;
+ char w2[MAXSWL];
+ const char * word = w;
+
+ // word reversing wrapper for complex prefixes
+ if (complexprefixes) {
+ strcpy(w2, w);
+ if (utf8) reverseword_utf(w2); else reverseword(w2);
+ word = w2;
+ }
+
rv = pAMgr->lookup(word);
while (rv) {
@@ -1338,7 +1379,6 @@ char * SuggestMgr::suggest_morph(const char * word)
0, 0, 100, 0, &r, NULL);
return (*result) ? mystrdup(line_uniq(delete_zeros(result))) : NULL;
-// return (*result) ? mystrdup(line_uniq(result)) : NULL;
}
char * SuggestMgr::suggest_morph_for_spelling_error(const char * word)
@@ -1361,26 +1401,137 @@ char * SuggestMgr::suggest_morph_for_spelling_error(const char * word)
int SuggestMgr::ngram(int n, char * s1, const char * s2, int uselen)
{
int nscore = 0;
- int l1 = strlen(s1);
- int l2 = strlen(s2);
int ns;
- for (int j=1;j<=n;j++) {
- ns = 0;
- for (int i=0;i<=(l1-j);i++) {
- char c = *(s1 + i + j);
- *(s1 + i + j) = '\0';
- if (strstr(s2,(s1+i))) ns++;
- *(s1 + i + j ) = c;
+ int l1;
+ int l2;
+
+ if (utf8) {
+ w_char su1[MAXSWL];
+ w_char su2[MAXSWL];
+ l1 = u8_u16(su1, MAXSWL, s1);
+ l2 = u8_u16(su2, MAXSWL, s2);
+ if (!l2) return 0;
+ // decapitalize dictionary word
+ if (complexprefixes) {
+ mkallsmall_utf(su2+l2-1, 1, utfconv);
+ } else {
+ mkallsmall_utf(su2, 1, utfconv);
+ }
+ for (int j = 1; j <= n; j++) {
+ ns = 0;
+ for (int i = 0; i <= (l1-j); i++) {
+ for (int l = 0; l <= (l2-j); l++) {
+ int k;
+ for (k = 0; (k < j); k++) {
+ w_char * c1 = su1 + i + k;
+ w_char * c2 = su2 + l + k;
+ if ((c1->l != c2->l) || (c1->h != c2->h)) break;
+ }
+ if (k == j) {
+ ns++;
+ break;
+ }
+ }
+ }
+ nscore = nscore + ns;
+ if (ns < 2) break;
+ }
+ } else {
+ char t[MAXSWL];
+ l1 = strlen(s1);
+ l2 = strlen(s2);
+ if (!l2) return 0;
+ strcpy(t, s2);
+ if (complexprefixes) {
+ *(t+l2-1) = csconv[((unsigned char)*(t+l2-1))].clower;
+ } else {
+ *t = csconv[((unsigned char)*t)].clower;
+ }
+ for (int j = 1; j <= n; j++) {
+ ns = 0;
+ for (int i = 0; i <= (l1-j); i++) {
+ char c = *(s1 + i + j);
+ *(s1 + i + j) = '\0';
+ if (strstr(t,(s1+i))) ns++;
+ *(s1 + i + j ) = c;
+ }
+ nscore = nscore + ns;
+ if (ns < 2) break;
}
- nscore = nscore + ns;
- if (ns < 2) break;
}
+
ns = 0;
if (uselen == NGRAM_LONGER_WORSE) ns = (l2-l1)-2;
if (uselen == NGRAM_ANY_MISMATCH) ns = abs(l2-l1)-2;
return (nscore - ((ns > 0) ? ns : 0));
}
+int SuggestMgr::equalfirstletter(char * s1, const char * s2) {
+ if (utf8) {
+ w_char su1[MAXSWL];
+ w_char su2[MAXSWL];
+ // decapitalize dictionary word
+ if (complexprefixes) {
+ int l1 = u8_u16(su1, MAXSWL, s1);
+ int l2 = u8_u16(su2, MAXSWL, s2);
+ mkallsmall_utf(su2+l2-1, 1, utfconv);
+ if (*((short *)su1+l1-1) == *((short *)su2+l2-1)) return 1;
+ } else {
+ u8_u16(su1, 1, s1);
+ u8_u16(su2, 1, s2);
+ mkallsmall_utf(su2, 1, utfconv);
+ if (*((short *)su1) == *((short *)su2)) return 1;
+ }
+ } else {
+ if (complexprefixes) {
+ int l1 = strlen(s1);
+ int l2 = strlen(s2);
+ if (*(s2+l1-1) == csconv[((unsigned char)*(s2+l2-1))].clower) return 1;
+ } else {
+ if (*s1 == csconv[((unsigned char)*s2)].clower) return 1;
+ }
+ }
+ return 0;
+}
+
+int SuggestMgr::commoncharacterpositions(char * s1, const char * s2) {
+ int num = 0;
+ if (utf8) {
+ w_char su1[MAXSWL];
+ w_char su2[MAXSWL];
+ int l1 = u8_u16(su1, MAXSWL, s1);
+ int l2 = u8_u16(su2, MAXSWL, s2);
+ // decapitalize dictionary word
+ if (complexprefixes) {
+ mkallsmall_utf(su2+l2-1, 1, utfconv);
+ char x[MAXSWL];
+ u16_u8(x, MAXSWL, su2, l2);
+ } else {
+ mkallsmall_utf(su2, 1, utfconv);
+ }
+ for (int i = 0; (i < l1) && (i < l2); i++)
+ if (((short *) su1)[i] == ((short *) su2)[i]) num++;
+ } else {
+ char t[MAXSWL];
+ strcpy(t, s2);
+ if (complexprefixes) {
+ int l = strlen(t);
+ *(t+l-1) = csconv[((unsigned char)*(t+l-1))].clower;
+ } else {
+ *t = csconv[((unsigned char)*t)].clower;
+ }
+ for (int i = 0; (*(s1+i) != 0) && (*(t+i) != 0); i++)
+ if (*(s1+i) == *(t+i)) num++;
+ }
+ return num;
+}
+
+int SuggestMgr::mystrlen(const char * word) {
+ if (utf8) {
+ w_char w[MAXSWL];
+ return u8_u16(w, MAXSWL, word);
+ } else return strlen(word);
+}
// sort in decreasing order of score
void SuggestMgr::bubblesort(char** rword, int* rsc, int n )
@@ -1404,3 +1555,66 @@ void SuggestMgr::bubblesort(char** rword, int* rsc, int n )
return;
}
+// longest common subsequence
+void SuggestMgr::lcs(const char * s, const char * s2, int * l1, int * l2, char ** result) {
+ int n, m;
+ w_char su[MAXSWL];
+ w_char su2[MAXSWL];
+ char * b;
+ char * c;
+ int i;
+ int j;
+ if (utf8) {
+ m = u8_u16(su, MAXSWL, s);
+ n = u8_u16(su2, MAXSWL, s2);
+ } else {
+ m = strlen(s);
+ n = strlen(s2);
+ }
+ c = (char *) malloc((m + 1) * (n + 1));
+ b = (char *) malloc((m + 1) * (n + 1));
+ for (i = 1; i <= m; i++) c[i*(n+1)] = 0;
+ for (j = 0; j <= n; j++) c[j] = 0;
+ for (i = 1; i <= m; i++) {
+ for (j = 1; j <= n; j++) {
+ if ((utf8) && (*((short *) su+i-1) == *((short *)su2+j-1))
+ || (!utf8) && ((*(s+i-1)) == (*(s2+j-1)))) {
+ c[i*(n+1) + j] = c[(i-1)*(n+1) + j-1]+1;
+ b[i*(n+1) + j] = LCS_UPLEFT;
+ } else if (c[(i-1)*(n+1) + j] >= c[i*(n+1) + j-1]) {
+ c[i*(n+1) + j] = c[(i-1)*(n+1) + j];
+ b[i*(n+1) + j] = LCS_UP;
+ } else {
+ c[i*(n+1) + j] = c[i*(n+1) + j-1];
+ b[i*(n+1) + j] = LCS_LEFT;
+ }
+ }
+ }
+ *result = b;
+ free(c);
+ *l1 = m;
+ *l2 = n;
+}
+
+int SuggestMgr::lcslen(const char * s, const char* s2) {
+ int m;
+ int n;
+ int i;
+ int j;
+ char * result;
+ int len = 0;
+ lcs(s, s2, &m, &n, &result);
+ i = m;
+ j = n;
+ while ((i != 0) && (j != 0)) {
+ if (result[i*(n+1) + j] == LCS_UPLEFT) {
+ len++;
+ i--;
+ j--;
+ } else if (result[i*(n+1) + j] == LCS_UP) {
+ i--;
+ } else j--;
+ }
+ if (result) free(result);
+ return len;
+}
diff --git a/src/hunspell/suggestmgr.hxx b/src/hunspell/suggestmgr.hxx
index 33558a0..d249d24 100644
--- a/src/hunspell/suggestmgr.hxx
+++ b/src/hunspell/suggestmgr.hxx
@@ -1,11 +1,14 @@
#ifndef _SUGGESTMGR_HXX_
#define _SUGGESTMGR_HXX_
-#define MAXSWL 100
-#define MAXACCENT 5
-#define MAX_ROOTS 10
-#define MAX_WORDS 500
-#define MAX_GUESS 10
+#define MAXSWL 300
+#define MAX_ROOTS 50
+#define MAX_WORDS 200
+#define MAX_GUESS 200
+#define MAXNGRAMSUGS 3
+
+#define MINTIMER 500
+#define MAXPLUSTIMER 500
#define NGRAM_IGNORE_LENGTH 0
#define NGRAM_LONGER_WORSE 1
@@ -15,6 +18,9 @@
#include "affixmgr.hxx"
#include "hashmgr.hxx"
#include "langnum.hxx"
+#include <time.h>
+
+enum { LCS_UP, LCS_LEFT, LCS_UPLEFT };
class SuggestMgr
{
@@ -24,12 +30,14 @@ class SuggestMgr
AffixMgr* pAMgr;
int maxSug;
- int cpdsuggest; // XXX not stateless
- int checknum;
struct cs_info * csconv;
+ struct unicode_info2 * utfconv;
+ int utf8;
int nosplitsugs;
- int nomapsugs;
-
+ int maxngramsugs;
+ int complexprefixes;
+
+
public:
SuggestMgr(const char * tryme, int maxn, AffixMgr *aptr);
~SuggestMgr();
@@ -39,33 +47,38 @@ public:
int suggest_auto(char*** slst, const char * word, int nsug);
int suggest_stems(char*** slst, const char * word, int nsug);
int suggest_pos_stems(char*** slst, const char * word, int nsug);
- int check(const char *, int);
- int check_forbidden(const char *, int);
char * suggest_morph(const char * word);
char * suggest_morph_for_spelling_error(const char * word);
private:
- int replchars(char**, const char *, int);
- int doubledsyllable(char**, const char *, int);
- int permuteaccent(char **, char *, int, int, int, replentry *, int);
- int forgotaccent(char **, const char *, int);
- int forgotchar(char **, const char *, int);
- int swapchar(char **, const char *, int);
- int extrachar(char **, const char *, int);
- int badchar(char **, const char *, int);
- int twowords(char **, const char *, int);
+ int check(const char *, int, int, int *, time_t *);
+ int check_forbidden(const char *, int);
+
+ int replchars(char**, const char *, int, int);
+ int doubledsyllable(char**, const char *, int, int);
+ int forgotchar(char **, const char *, int, int);
+ int swapchar(char **, const char *, int, int);
+ int extrachar(char **, const char *, int, int);
+ int badchar(char **, const char *, int, int);
+ int twowords(char **, const char *, int, int);
int fixstems(char **, const char *, int);
- int forgotchar_utf(char**, const w_char *, int wl, int);
- int extrachar_utf(char**, const w_char *, int wl, int);
- int badchar_utf(char **, const w_char *, int wl, int);
- int swapchar_utf(char **, const w_char *, int wl, int);
+ int forgotchar_utf(char**, const w_char *, int wl, int, int);
+ int extrachar_utf(char**, const w_char *, int wl, int, int);
+ int badchar_utf(char **, const w_char *, int wl, int, int);
+ int swapchar_utf(char **, const w_char *, int wl, int, int);
- int mapchars(char**, const char *, int);
- int map_related(const char *, int, char ** wlst, int, const mapentry*, int);
+ int mapchars(char**, const char *, int, int);
+ int map_related(const char *, int, char ** wlst, int, const mapentry*, int, int *, time_t *);
+ int map_related_utf(w_char *, int, int, char ** wlst, int, const mapentry*, int, int *, time_t *);
int ngram(int n, char * s1, const char * s2, int uselen);
+ int mystrlen(const char * word);
+ int equalfirstletter(char * s1, const char * s2);
+ int commoncharacterpositions(char * s1, const char * s2);
void bubblesort( char ** rwd, int * rsc, int n);
+ void lcs(const char * s, const char * s2, int * l1, int * l2, char ** result);
+ int lcslen(const char * s, const char* s2);
};
diff --git a/src/tools/Makefile.am b/src/tools/Makefile.am
index 85b27f2..fa93bd2 100644
--- a/src/tools/Makefile.am
+++ b/src/tools/Makefile.am
@@ -1,9 +1,13 @@
-bin_PROGRAMS=unmunch hunspell hunmorph hunstem
+bin_PROGRAMS=munch unmunch example hunspell hunmorph hunstem
INCLUDES=-I${top_srcdir}/src/hunspell -I${top_srcdir}/src/parsers
+munch_SOURCES=munch.c
unmunch_SOURCES=unmunch.c
-include_HEADERS=unmunch.h
+include_HEADERS=munch.h unmunch.h
+
+example_SOURCES=example.cxx
+example_LDADD = ../hunspell/libhunspell.a
hunspell_SOURCES=hunspell.cxx
hunspell_LDADD = @LIBINTL@ ../hunspell/libhunspell.a \
diff --git a/src/tools/Makefile.in b/src/tools/Makefile.in
index 8873b5f..3db7ca6 100644
--- a/src/tools/Makefile.in
+++ b/src/tools/Makefile.in
@@ -15,7 +15,7 @@
@SET_MAKE@
-SOURCES = $(hunmorph_SOURCES) $(hunspell_SOURCES) $(hunstem_SOURCES) $(unmunch_SOURCES)
+SOURCES = $(example_SOURCES) $(hunmorph_SOURCES) $(hunspell_SOURCES) $(hunstem_SOURCES) $(munch_SOURCES) $(unmunch_SOURCES)
srcdir = @srcdir@
top_srcdir = @top_srcdir@
@@ -39,8 +39,8 @@ PRE_UNINSTALL = :
POST_UNINSTALL = :
build_triplet = @build@
host_triplet = @host@
-bin_PROGRAMS = unmunch$(EXEEXT) hunspell$(EXEEXT) hunmorph$(EXEEXT) \
- hunstem$(EXEEXT)
+bin_PROGRAMS = munch$(EXEEXT) unmunch$(EXEEXT) example$(EXEEXT) \
+ hunspell$(EXEEXT) hunmorph$(EXEEXT) hunstem$(EXEEXT)
subdir = src/tools
DIST_COMMON = $(include_HEADERS) $(srcdir)/Makefile.am \
$(srcdir)/Makefile.in
@@ -59,6 +59,9 @@ CONFIG_CLEAN_FILES =
am__installdirs = "$(DESTDIR)$(bindir)" "$(DESTDIR)$(includedir)"
binPROGRAMS_INSTALL = $(INSTALL_PROGRAM)
PROGRAMS = $(bin_PROGRAMS)
+am_example_OBJECTS = example.$(OBJEXT)
+example_OBJECTS = $(am_example_OBJECTS)
+example_DEPENDENCIES = ../hunspell/libhunspell.a
am_hunmorph_OBJECTS = hunmorph.$(OBJEXT)
hunmorph_OBJECTS = $(am_hunmorph_OBJECTS)
hunmorph_DEPENDENCIES = ../hunspell/libhunspell.a
@@ -69,6 +72,9 @@ hunspell_DEPENDENCIES = ../hunspell/libhunspell.a \
am_hunstem_OBJECTS = hunstem.$(OBJEXT)
hunstem_OBJECTS = $(am_hunstem_OBJECTS)
hunstem_DEPENDENCIES = ../hunspell/libhunspell.a
+am_munch_OBJECTS = munch.$(OBJEXT)
+munch_OBJECTS = $(am_munch_OBJECTS)
+munch_LDADD = $(LDADD)
am_unmunch_OBJECTS = unmunch.$(OBJEXT)
unmunch_OBJECTS = $(am_unmunch_OBJECTS)
unmunch_LDADD = $(LDADD)
@@ -84,10 +90,11 @@ CXXCOMPILE = $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \
CXXLD = $(CXX)
CXXLINK = $(CXXLD) $(AM_CXXFLAGS) $(CXXFLAGS) $(AM_LDFLAGS) $(LDFLAGS) \
-o $@
-SOURCES = $(hunmorph_SOURCES) $(hunspell_SOURCES) $(hunstem_SOURCES) \
+SOURCES = $(example_SOURCES) $(hunmorph_SOURCES) $(hunspell_SOURCES) \
+ $(hunstem_SOURCES) $(munch_SOURCES) $(unmunch_SOURCES)
+DIST_SOURCES = $(example_SOURCES) $(hunmorph_SOURCES) \
+ $(hunspell_SOURCES) $(hunstem_SOURCES) $(munch_SOURCES) \
$(unmunch_SOURCES)
-DIST_SOURCES = $(hunmorph_SOURCES) $(hunspell_SOURCES) \
- $(hunstem_SOURCES) $(unmunch_SOURCES)
am__vpath_adj_setup = srcdirstrip=`echo "$(srcdir)" | sed 's|.|.|g'`;
am__vpath_adj = case $$p in \
$(srcdir)/*) f=`echo "$$p" | sed "s|^$$srcdirstrip/||"`;; \
@@ -212,8 +219,11 @@ sharedstatedir = @sharedstatedir@
sysconfdir = @sysconfdir@
target_alias = @target_alias@
INCLUDES = -I${top_srcdir}/src/hunspell -I${top_srcdir}/src/parsers
+munch_SOURCES = munch.c
unmunch_SOURCES = unmunch.c
-include_HEADERS = unmunch.h
+include_HEADERS = munch.h unmunch.h
+example_SOURCES = example.cxx
+example_LDADD = ../hunspell/libhunspell.a
hunspell_SOURCES = hunspell.cxx
hunspell_LDADD = @LIBINTL@ ../hunspell/libhunspell.a \
../parsers/libparsers.a @CURSESLIB@ @READLINELIB@
@@ -279,6 +289,9 @@ uninstall-binPROGRAMS:
clean-binPROGRAMS:
-test -z "$(bin_PROGRAMS)" || rm -f $(bin_PROGRAMS)
+example$(EXEEXT): $(example_OBJECTS) $(example_DEPENDENCIES)
+ @rm -f example$(EXEEXT)
+ $(CXXLINK) $(example_LDFLAGS) $(example_OBJECTS) $(example_LDADD) $(LIBS)
hunmorph$(EXEEXT): $(hunmorph_OBJECTS) $(hunmorph_DEPENDENCIES)
@rm -f hunmorph$(EXEEXT)
$(CXXLINK) $(hunmorph_LDFLAGS) $(hunmorph_OBJECTS) $(hunmorph_LDADD) $(LIBS)
@@ -288,6 +301,9 @@ hunspell$(EXEEXT): $(hunspell_OBJECTS) $(hunspell_DEPENDENCIES)
hunstem$(EXEEXT): $(hunstem_OBJECTS) $(hunstem_DEPENDENCIES)
@rm -f hunstem$(EXEEXT)
$(CXXLINK) $(hunstem_LDFLAGS) $(hunstem_OBJECTS) $(hunstem_LDADD) $(LIBS)
+munch$(EXEEXT): $(munch_OBJECTS) $(munch_DEPENDENCIES)
+ @rm -f munch$(EXEEXT)
+ $(LINK) $(munch_LDFLAGS) $(munch_OBJECTS) $(munch_LDADD) $(LIBS)
unmunch$(EXEEXT): $(unmunch_OBJECTS) $(unmunch_DEPENDENCIES)
@rm -f unmunch$(EXEEXT)
$(LINK) $(unmunch_LDFLAGS) $(unmunch_OBJECTS) $(unmunch_LDADD) $(LIBS)
@@ -298,9 +314,11 @@ mostlyclean-compile:
distclean-compile:
-rm -f *.tab.c
+@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/example.Po@am__quote@
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hunmorph.Po@am__quote@
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hunspell.Po@am__quote@
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hunstem.Po@am__quote@
+@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/munch.Po@am__quote@
@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/unmunch.Po@am__quote@
.c.o:
diff --git a/src/tools/example.cxx b/src/tools/example.cxx
new file mode 100644
index 0000000..0f53927
--- /dev/null
+++ b/src/tools/example.cxx
@@ -0,0 +1,89 @@
+#include <cstring>
+#include <cstdlib>
+#include <cstdio>
+
+#include "hunspell.hxx"
+
+extern char * mystrdup(const char * s);
+
+using namespace std;
+
+int
+main(int argc, char** argv)
+{
+
+ char * af;
+ char * df;
+ char * wtc;
+ FILE* wtclst;
+
+ /* first parse the command line options */
+ /* arg1 - affix file, arg2 dictionary file, arg3 - file of words to check */
+
+ if (argv[1]) {
+ af = mystrdup(argv[1]);
+ } else {
+ fprintf(stderr,"correct syntax is:\n");
+ fprintf(stderr,"example affix_file dictionary_file file_of_words_to_check\n");
+ exit(1);
+ }
+ if (argv[2]) {
+ df = mystrdup(argv[2]);
+ } else {
+ fprintf(stderr,"correct syntax is:\n");
+ fprintf(stderr,"example affix_file dictionary_file file_of_words_to_check\n");
+ exit(1);
+ }
+ if (argv[3]) {
+ wtc = mystrdup(argv[3]);
+ } else {
+ fprintf(stderr,"correct syntax is:\n");
+ fprintf(stderr,"example affix_file dictionary_file file_of_words_to_check\n");
+ exit(1);
+ }
+
+
+ /* open the words to check list */
+ wtclst = fopen(wtc,"r");
+ if (!wtclst) {
+ fprintf(stderr,"Error - could not open file of words to check\n");
+ exit(1);
+ }
+
+
+ Hunspell * pMS= new Hunspell(af,df);
+
+ int k;
+ int dp;
+ char buf[101];
+
+ while(fgets(buf,100,wtclst)) {
+ k = strlen(buf);
+ *(buf + k - 1) = '\0';
+ dp = pMS->spell(buf);
+ if (dp) {
+ fprintf(stdout,"\"%s\" is okay\n",buf);
+ fprintf(stdout,"\n");
+ } else {
+ fprintf(stdout,"\"%s\" is incorrect!\n",buf);
+ fprintf(stdout," suggestions:\n");
+ char ** wlst;
+ int ns = pMS->suggest(&wlst,buf);
+ for (int i=0; i < ns; i++) {
+ fprintf(stdout," ...\"%s\"\n",wlst[i]);
+ free(wlst[i]);
+ }
+ fprintf(stdout,"\n");
+ free(wlst);
+ }
+ }
+
+ delete pMS;
+ fclose(wtclst);
+ free(wtc);
+ free(df);
+ free(af);
+
+ return 0;
+}
+
diff --git a/src/tools/unmunch.c b/src/tools/munch.c
similarity index 53%
copy from src/tools/unmunch.c
copy to src/tools/munch.c
index 673a8c7..d695b04 100644
--- a/src/tools/unmunch.c
+++ b/src/tools/munch.c
@@ -1,6 +1,4 @@
-/* Un-munch a root word list with affix tags
- * to recreate the original word list
- */
+/* Munch a word list and generate a smaller root word list with affixes*/
#include <ctype.h>
#include <string.h>
@@ -16,37 +14,42 @@
#endif
#include <sys/mman.h>
-#include "unmunch.h"
-
+#include "munch.h"
int main(int argc, char** argv)
{
- int i;
- int al, wl;
+ int i, j, k, n;
+ int rl, p , nwl;
+ int al;
FILE * wrdlst;
FILE * afflst;
- char *wf, *af;
+ char *nword, *wf, *af;
+ char as[(MAX_PREFIXES + MAX_SUFFIXES)];
char * ap;
- char ts[MAX_LN_LEN];
+
+ struct hentry * ep;
+ struct hentry * ep1;
+ struct affent * pfxp;
+ struct affent * sfxp;
/* first parse the command line options */
- /* arg1 - munched wordlist, arg2 - affix file */
+ /* arg1 - wordlist, arg2 - affix file */
if (argv[1]) {
wf = mystrdup(argv[1]);
} else {
fprintf(stderr,"correct syntax is:\n");
- fprintf(stderr,"unmunch dic_file affix_file\n");
+ fprintf(stderr,"munch word_list_file affix_file\n");
exit(1);
}
if (argv[2]) {
af = mystrdup(argv[2]);
} else {
fprintf(stderr,"correct syntax is:\n");
- fprintf(stderr,"unmunch dic_file affix_file\n");
+ fprintf(stderr,"munch word_list_file affix_file\n");
exit(1);
}
@@ -77,54 +80,132 @@ int main(int argc, char** argv)
exit(1);
}
- /* skip over the hash table size */
- if (! fgets(ts, MAX_LN_LEN-1,wrdlst)) return 2;
- mychomp(ts);
-
- while (fgets(ts,MAX_LN_LEN-1,wrdlst)) {
- mychomp(ts);
- /* split each line into word and affix char strings */
- ap = strchr(ts,'/');
- if (ap) {
- *ap = '\0';
- ap++;
- al = strlen(ap);
- } else {
- al = 0;
- ap = NULL;
- }
-
- wl = strlen(ts);
+ if (load_tables(wrdlst)) {
+ fprintf(stderr,"Error building hash tables\n");
+ exit(1);
+ }
+ fclose(wrdlst);
- numwords = 0;
- wlist[numwords].word = mystrdup(ts);
- wlist[numwords].pallow = 0;
- numwords++;
-
- if (al)
- expand_rootword(ts,wl,ap,al);
+ for (i=0; i< tablesize; i++) {
+ ep = &tableptr[i];
+ if (ep->word == NULL) continue;
+ for ( ; ep != NULL; ep = ep->next) {
+ numroots = 0;
+ aff_chk(ep->word,strlen(ep->word));
+ if (numroots) {
+ /* now there might be a number of combinations */
+ /* of prefixes and suffixes that might match this */
+ /* word. So how to choose? As a first shot look */
+ /* for the shortest remaining root word to */
+ /* to maximize the combinatorial power */
+
+ /* but be careful, do not REQUIRE a specific combination */
+ /* of a prefix and a suffix to generate the word since */
+ /* that violates the rule that the root word with just */
+ /* the prefix or just the suffix must also exist in the */
+ /* wordlist as well */
+
+ /* in fact because of the cross product issue, this not a */
+ /* simple choice since some combinations of previous */
+ /* prefixes and new suffixes may not be valid. */
+ /* The only way to know is to simply try them all */
- for (i=0; i < numwords; i++) {
- fprintf(stdout,"%s\n",wlist[i].word);
- free(wlist[i].word);
- wlist[i].word = NULL;
- wlist[i].pallow = 0;
- }
+ rl = 1000;
+ p = -1;
+
+ for (j = 0; j < numroots; j++){
+
+ /* first collect the root word info and build up */
+ /* the potential new affix string */
+ nword = (roots[j].hashent)->word;
+ nwl = strlen(nword);
+ *as = '\0';
+ al = 0;
+ ap = as;
+ if (roots[j].prefix) *ap++ = (roots[j].prefix)->achar;
+ if (roots[j].suffix) *ap++ = (roots[j].suffix)->achar;
+ if ((roots[j].hashent)->affstr) {
+ strcpy(ap,(roots[j].hashent)->affstr);
+ } else {
+ *ap = '\0';
+ }
+ al =strlen(as);
+
+ /* now expand the potential affix string to generate */
+ /* all legal words and make sure they all exist in the */
+ /* word list */
+ numwords = 0;
+ wlist[numwords].word = mystrdup(nword);
+ wlist[numwords].pallow = 0;
+ numwords++;
+ n = 0;
+ if (al)
+ expand_rootword(nword,nwl,as,al);
+ for (k=0; k<numwords; k++) {
+ if (lookup(wlist[k].word)) n++;
+ free(wlist[k].word);
+ wlist[k].word = NULL;
+ wlist[k].pallow = 0;
+ }
+ /* if all exist in word list then okay */
+ if (n == numwords) {
+ if (nwl < rl) {
+ rl = nwl;
+ p = j;
+ }
+ }
+ }
+ if (p != -1) {
+ ep1 = roots[p].hashent;
+ pfxp = roots[p].prefix;
+ sfxp = roots[p].suffix;
+ ep1->keep = 1;
+ if (pfxp != NULL) add_affix_char(ep1,pfxp->achar);
+ if (sfxp != NULL) add_affix_char(ep1,sfxp->achar);
+ } else {
+ ep->keep = 1;
+ }
+ } else {
+ ep->keep = 1;
+ }
+ }
}
- fclose(wrdlst);
+ /* now output only the words to keep along with affixes info */
+ /* first count how many words that is */
+ k = 0;
+ for (i=0; i< tablesize; i++) {
+ ep = &tableptr[i];
+ if (ep->word == NULL) continue;
+ for ( ; ep != NULL; ep = ep->next) {
+ if (ep->keep > 0) k++;
+ }
+ }
+ fprintf(stdout,"%d\n",k);
+
+ for (i=0; i< tablesize; i++) {
+ ep = &tableptr[i];
+ if (ep->word == NULL) continue;
+ for ( ; ep != NULL; ep = ep->next) {
+ if (ep->keep > 0) {
+ if (ep->affstr != NULL) {
+ fprintf(stdout,"%s/%s\n",ep->word,ep->affstr);
+ } else {
+ fprintf(stdout,"%s\n",ep->word);
+ }
+ }
+ }
+ }
return 0;
}
-
-
void parse_aff_file(FILE * afflst)
{
int i, j;
- int numents=0;
- char achar='\0';
+ int numents = 0;
+ char achar = '\0';
short ff=0;
char ft;
struct affent * ptr= NULL;
@@ -140,8 +221,8 @@ void parse_aff_file(FILE * afflst)
if (ft != ' ') {
char * tp = line;
char * piece;
- ff = 0;
i = 0;
+ ff = 0;
while ((piece=mystrsep(&tp,' '))) {
if (*piece != '\0') {
switch(i) {
@@ -210,12 +291,12 @@ void parse_aff_file(FILE * afflst)
if (ft == 'P') {
ptable[numpfx].aep = ptr;
ptable[numpfx].num = numents;
- fprintf(stderr,"ptable %d num is %d flag %c\n",numpfx,ptable[numpfx].num,ptr->achar);
+ fprintf(stderr,"ptable %d num is %d\n",numpfx,ptable[numpfx].num);
numpfx++;
} else {
stable[numsfx].aep = ptr;
stable[numsfx].num = numents;
- fprintf(stderr,"stable %d num is %d flag %c\n",numsfx,stable[numsfx].num,ptr->achar);
+ fprintf(stderr,"stable %d num is %d\n",numsfx,stable[numsfx].num);
numsfx++;
}
ptr = NULL;
@@ -315,6 +396,252 @@ void encodeit(struct affent * ptr, char * cs)
+/* search for a prefix */
+void pfx_chk (const char * word, int len, struct affent* ep, int num)
+{
+ struct affent * aent;
+ int cond;
+ int tlen;
+ struct hentry * hent;
+ unsigned char * cp;
+ int i;
+ char tword[MAX_WD_LEN];
+
+ for (aent = ep, i = num; i > 0; aent++, i--) {
+
+ tlen = len - aent->appndl;
+
+ if (tlen > 0 && (aent->appndl == 0 ||
+ strncmp(aent->appnd, word, aent->appndl) == 0)
+ && tlen + aent->stripl >= aent->numconds) {
+
+ if (aent->stripl) strcpy (tword, aent->strip);
+ strcpy((tword + aent->stripl), (word + aent->appndl));
+
+ /* now go through the conds and make sure they all match */
+ cp = (unsigned char *) tword;
+ for (cond = 0; cond < aent->numconds; cond++) {
+ if ((aent->conds[*cp++] & (1 << cond)) == 0)
+ break;
+ }
+
+ if (cond >= aent->numconds) {
+ tlen += aent->stripl;
+ if ((hent = lookup(tword)) != NULL) {
+ if (numroots < MAX_ROOTS) {
+ roots[numroots].hashent = hent;
+ roots[numroots].prefix = aent;
+ roots[numroots].suffix = NULL;
+ numroots++;
+ }
+ }
+ }
+ }
+ }
+}
+
+
+
+void suf_chk (const char * word, int len, struct affent * ep,
+ int num, struct affent * pfxent, int cpflag)
+{
+ struct affent * aent;
+ int tlen;
+ int cond;
+ struct hentry * hent;
+ unsigned char * cp;
+ int i;
+ char tword[MAX_WD_LEN];
+
+ for (aent = ep, i = num; i > 0; aent++, i--) {
+
+ if ((cpflag & XPRODUCT) != 0 && (aent->xpflg & XPRODUCT) == 0)
+ continue;
+
+ tlen = len - aent->appndl;
+ if (tlen > 0 && (aent->appndl == 0 ||
+ strcmp(aent->appnd, (word + tlen)) == 0)
+ && tlen + aent->stripl >= aent->numconds) {
+
+ strcpy (tword, word);
+ cp = (unsigned char *) (tword + tlen);
+ if (aent->stripl) {
+ strcpy ((char *)cp, aent->strip);
+ tlen += aent->stripl;
+ cp = (unsigned char *)(tword + tlen);
+ } else *cp = '\0';
+
+ for (cond = aent->numconds; --cond >= 0; ) {
+ if ((aent->conds[*--cp] & (1 << cond)) == 0) break;
+ }
+ if (cond < 0) {
+ if ((hent = lookup(tword)) != NULL) {
+ if (numroots < MAX_ROOTS) {
+ roots[numroots].hashent = hent;
+ roots[numroots].prefix = pfxent;
+ roots[numroots].suffix = aent;
+ numroots++;
+ }
+ }
+ }
+ }
+ }
+}
+
+
+
+void aff_chk (const char * word, int len)
+{
+ int i;
+ int j;
+ int nh=0;
+ char * nword;
+ int nwl;
+
+ if (len < 4) return;
+
+ for (i=0; i < numpfx; i++) {
+ pfx_chk(word, len, ptable[i].aep, ptable[i].num);
+ }
+
+ nh = numroots;
+
+ if (nh > 0) {
+ for (j=0;j<nh;j++){
+ if (roots[j].prefix->xpflg & XPRODUCT) {
+ nword = mystrdup((roots[j].hashent)->word);
+ nwl = strlen(nword);
+ for (i=0; i < numsfx; i++) {
+ suf_chk(nword,nwl,stable[i].aep, stable[i].num, roots[j].prefix, XPRODUCT);
+ }
+ free(nword);
+ }
+ }
+ }
+ for (i=0; i < numsfx; i++) {
+ suf_chk(word, len, stable[i].aep, stable[i].num, NULL, 0);
+ }
+}
+
+
+
+/* lookup a root word in the hashtable */
+
+struct hentry * lookup(const char *word)
+{
+ struct hentry * dp;
+ dp = &tableptr[hash(word)];
+ if (dp->word == NULL) return NULL;
+ for ( ; dp != NULL; dp = dp->next) {
+ if (strcmp(word,dp->word) == 0) return dp;
+ }
+ return NULL;
+}
+
+
+
+/* add a word to the hash table */
+
+int add_word(char * word)
+{
+ int i;
+ struct hentry * dp;
+ struct hentry * hp = (struct hentry *) malloc (sizeof(struct hentry));
+
+ hp->word = word;
+ hp->affstr = NULL;
+ hp->keep = 0;
+ hp->next = NULL;
+
+ i = hash(word);
+ dp = &tableptr[i];
+
+ if (dp->word == NULL) {
+ *dp = *hp;
+ free(hp);
+ } else {
+ while (dp->next != NULL) dp=dp->next;
+ dp->next = hp;
+ }
+ return 0;
+}
+
+
+
+/* load a word list and build a hash table on the fly */
+
+int load_tables(FILE * wdlst)
+{
+ char * ap;
+ char ts[MAX_LN_LEN];
+
+ /* first read the first line of file to get hash table size */
+ if (! fgets(ts, MAX_LN_LEN-1,wdlst)) return 2;
+ mychomp(ts);
+ tablesize = atoi(ts);
+ tablesize = tablesize + 5;
+ if ((tablesize %2) == 0) tablesize++;
+
+ /* allocate the hash table */
+ tableptr = (struct hentry *) calloc(tablesize, sizeof(struct hentry));
+ if (! tableptr) return 3;
+
+ /* loop thorugh all words on much list and add to hash
+ * table and store away word and affix strings in tmpfile
+ */
+
+ while (fgets(ts,MAX_LN_LEN-1,wdlst)) {
+ mychomp(ts);
+ ap = mystrdup(ts);
+ add_word(ap);
+
+ }
+ return 0;
+}
+
+
+/* the hash function is a simple load and rotate
+ * algorithm borrowed
+ */
+
+int hash(const char * word)
+{
+ int i;
+ long hv = 0;
+ for (i=0; i < 4 && *word != 0; i++)
+ hv = (hv << 8) | (*word++);
+ while (*word != 0) {
+ ROTATE(hv,ROTATE_LEN);
+ hv ^= (*word++);
+ }
+ return (unsigned long) hv % tablesize;
+}
+
+
+void add_affix_char(struct hentry * ep, char ac)
+{
+ int al;
+ int i;
+ char * tmp;
+ if (ep->affstr == NULL) {
+ ep->affstr = (char *) malloc(2*sizeof(char));
+ *(ep->affstr) = ac;
+ *((ep->affstr)+1) = '\0';
+ return;
+ }
+ al = strlen(ep->affstr);
+ for (i=0; i< al; i++)
+ if (ac == (ep->affstr)[i]) return;
+ tmp = calloc((al+2),sizeof(char));
+ memcpy(tmp,ep->affstr,(al+1));
+ *(tmp+al) = ac;
+ *(tmp+al+1)='\0';
+ free(ep->affstr);
+ ep->affstr = tmp;
+ return;
+}
+
+
/* add a prefix to word */
void pfx_add (const char * word, int len, struct affent* ep, int num)
{
@@ -446,6 +773,7 @@ int expand_rootword(const char * ts, int wl, const char * ap, int al)
}
+
/* strip strings into token based on single char delimiter
* acts like strsep() but only uses a delim char and not
* a delim string
diff --git a/src/tools/munch.h b/src/tools/munch.h
new file mode 100644
index 0000000..8dc7aea
--- /dev/null
+++ b/src/tools/munch.h
@@ -0,0 +1,121 @@
+/* munch header file */
+
+#define MAX_LN_LEN 200
+#define MAX_WD_LEN 200
+#define MAX_PREFIXES 256
+#define MAX_SUFFIXES 256
+#define MAX_ROOTS 20
+#define MAX_WORDS 5000
+
+#define ROTATE_LEN 5
+
+#define ROTATE(v,q) \
+ (v) = ((v) << (q)) | (((v) >> (32 - q)) & ((1 << (q))-1));
+
+#define SET_SIZE 256
+
+#define XPRODUCT (1 << 0)
+
+/* the affix table entry */
+
+struct affent
+{
+ char * appnd;
+ char * strip;
+ short appndl;
+ short stripl;
+ char achar;
+ char xpflg;
+ short numconds;
+ char conds[SET_SIZE];
+};
+
+
+struct affixptr
+{
+ struct affent * aep;
+ int num;
+};
+
+/* the prefix and suffix table */
+int numpfx; /* Number of prefixes in table */
+int numsfx; /* Number of suffixes in table */
+
+/* the prefix table */
+struct affixptr ptable[MAX_PREFIXES];
+
+/* the suffix table */
+struct affixptr stable[MAX_SUFFIXES];
+
+
+/* data structure to store results of lookups */
+struct matches
+{
+ struct hentry * hashent; /* hash table entry */
+ struct affent * prefix; /* Prefix used, or NULL */
+ struct affent * suffix; /* Suffix used, or NULL */
+};
+
+int numroots; /* number of root words found */
+struct matches roots[MAX_ROOTS]; /* list of root words found */
+
+/* hashing stuff */
+
+struct hentry
+{
+ char * word;
+ char * affstr;
+ struct hentry * next;
+ int keep;
+};
+
+
+int tablesize;
+struct hentry * tableptr;
+
+/* unmunch stuff */
+
+int numwords; /* number of words found */
+struct dwords
+{
+ char * word;
+ int pallow;
+};
+
+struct dwords wlist[MAX_WORDS]; /* list words found */
+
+
+/* the routines */
+
+void parse_aff_file(FILE* afflst);
+
+void encodeit(struct affent * ptr, char * cs);
+
+int load_tables(FILE * wrdlst);
+
+int hash(const char *);
+
+int add_word(char *);
+
+struct hentry * lookup(const char *);
+
+void aff_chk (const char * word, int len);
+
+void pfx_chk (const char * word, int len, struct affent* ep, int num);
+
+void suf_chk (const char * word, int len, struct affent * ep, int num,
+ struct affent * pfxent, int cpflag);
+
+void add_affix_char(struct hentry * hent, char ac);
+
+int expand_rootword(const char *, int, const char*, int);
+
+void pfx_add (const char * word, int len, struct affent* ep, int num);
+
+void suf_add (const char * word, int len, struct affent * ep, int num);
+
+char * mystrsep(char ** stringp, const char delim);
+
+char * mystrdup(const char * s);
+
+void mychomp(char * s);
diff --git a/src/tools/unmunch.c b/src/tools/unmunch.c
index 673a8c7..12725c3 100644
--- a/src/tools/unmunch.c
+++ b/src/tools/unmunch.c
@@ -330,7 +330,9 @@ void pfx_add (const char * word, int len, struct affent* ep, int num)
for (aent = ep, i = num; i > 0; aent++, i--) {
/* now make sure all conditions match */
- if ((len > aent->stripl) && (len >= aent->numconds)) {
+ if ((len > aent->stripl) && (len >= aent->numconds) &&
+ ((aent->stripl == 0) ||
+ (strncmp(aent->strip, word, aent->stripl) == 0))) {
cp = (unsigned char *) word;
for (cond = 0; cond < aent->numconds; cond++) {
@@ -377,7 +379,9 @@ void suf_add (const char * word, int len, struct affent * ep, int num)
* then strip off strip string and add suffix
*/
- if ((len > aent->stripl) && (len >= aent->numconds)) {
+ if ((len > aent->stripl) && (len >= aent->numconds) &&
+ ((aent->stripl == 0) ||
+ (strcmp(aent->strip, word + len - aent->stripl - 1) == 0))) {
cp = (unsigned char *) (word + len);
for (cond = aent->numconds; --cond >= 0; ) {
if ((aent->conds[*--cp] & (1 << cond)) == 0) break;
@@ -492,7 +496,7 @@ char * mystrdup(const char * s)
void mychomp(char * s)
{
int k = strlen(s);
- if (k > 0) *(s+k-1) = '\0';
+ if ((k > 0) && (*(s+k-1) == '\n')) *(s+k-1) = '\0';
if ((k > 1) && (*(s+k-2) == '\r')) *(s+k-2) = '\0';
}
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 71ce86d..47277a4 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -7,8 +7,11 @@ flagnum.test \
forbiddenword.test \
pseudoroot.test \
pseudoroot2.test \
+pseudoroot3.test \
+pseudoroot4.test \
circumfix.test \
fogemorpheme.test \
+complexprefixes.test \
conditionalprefix.test \
zeroaffix.test \
utf8.test \
@@ -57,6 +60,10 @@ pseudoroot3.dic \
pseudoroot3.good \
pseudoroot3.test \
pseudoroot3.wrong \
+pseudoroot4.aff \
+pseudoroot4.dic \
+pseudoroot4.good \
+pseudoroot4.test \
pseudoroot.aff \
pseudoroot.dic \
pseudoroot.good \
@@ -109,6 +116,7 @@ germansharpsutf.wrong \
conditionalprefix.aff \
conditionalprefix.dic \
conditionalprefix.good \
+conditionalprefix.morph \
conditionalprefix.test \
conditionalprefix.wrong \
flaglong.aff \
@@ -122,4 +130,9 @@ flagnum.test \
flag.aff \
flag.dic \
flag.good \
-flag.test
+flag.test \
+complexprefixes.aff \
+complexprefixes.dic \
+complexprefixes.good \
+complexprefixes.wrong \
+complexprefixes.test
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 2bb18ac..079d549 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -171,8 +171,11 @@ flagnum.test \
forbiddenword.test \
pseudoroot.test \
pseudoroot2.test \
+pseudoroot3.test \
+pseudoroot4.test \
circumfix.test \
fogemorpheme.test \
+complexprefixes.test \
conditionalprefix.test \
zeroaffix.test \
utf8.test \
@@ -218,6 +221,10 @@ pseudoroot3.dic \
pseudoroot3.good \
pseudoroot3.test \
pseudoroot3.wrong \
+pseudoroot4.aff \
+pseudoroot4.dic \
+pseudoroot4.good \
+pseudoroot4.test \
pseudoroot.aff \
pseudoroot.dic \
pseudoroot.good \
@@ -270,6 +277,7 @@ germansharpsutf.wrong \
conditionalprefix.aff \
conditionalprefix.dic \
conditionalprefix.good \
+conditionalprefix.morph \
conditionalprefix.test \
conditionalprefix.wrong \
flaglong.aff \
@@ -283,7 +291,12 @@ flagnum.test \
flag.aff \
flag.dic \
flag.good \
-flag.test
+flag.test \
+complexprefixes.aff \
+complexprefixes.dic \
+complexprefixes.good \
+complexprefixes.wrong \
+complexprefixes.test
all: all-am
diff --git a/tests/complexprefixes.aff b/tests/complexprefixes.aff
new file mode 100644
index 0000000..7ddb497
--- /dev/null
+++ b/tests/complexprefixes.aff
@@ -0,0 +1,9 @@
+# set twofold prefix stripping
+# Coptic example by Moheb Mekhaiel
+COMPLEXPREFIXES
+
+PFX A Y 1
+PFX A 0 tek .
+
+PFX B Y 1
+PFX B 0 met/A .
diff --git a/tests/complexprefixes.dic b/tests/complexprefixes.dic
new file mode 100644
index 0000000..2618c7c
--- /dev/null
+++ b/tests/complexprefixes.dic
@@ -0,0 +1,3 @@
+1
+ouro/B
+
diff --git a/tests/complexprefixes.good b/tests/complexprefixes.good
new file mode 100644
index 0000000..eed87a7
--- /dev/null
+++ b/tests/complexprefixes.good
@@ -0,0 +1,3 @@
+ouro
+metouro
+tekmetouro
diff --git a/tests/complexprefixes.test b/tests/complexprefixes.test
new file mode 100755
index 0000000..89ca772
--- /dev/null
+++ b/tests/complexprefixes.test
@@ -0,0 +1,4 @@
+#!/bin/sh
+DIR="`dirname $0`"
+NAME="`basename $0 .test`"
+$DIR/test_hunmorph $NAME
diff --git a/tests/complexprefixes.wrong b/tests/complexprefixes.wrong
new file mode 100644
index 0000000..fb1c8b4
--- /dev/null
+++ b/tests/complexprefixes.wrong
@@ -0,0 +1,2 @@
+tekouro
+mettekouro
diff --git a/tests/conditionalprefix.morph b/tests/conditionalprefix.morph
new file mode 100644
index 0000000..3d67eb9
--- /dev/null
+++ b/tests/conditionalprefix.morph
@@ -0,0 +1,14 @@
+> drink
+drink[verb]
+drink[noun]
+> drinks
+drink[verb]+3SGV
+drink[noun]+PL
+> drinkable
+drink[verb]+DER_V_ADJ_ABLE
+> drinkables
+drink[verb]+DER_V_ADJ_ABLE+PL
+> undrinkable
+prefix_un+drink[verb]+DER_V_ADJ_ABLE
+> undrinkables
+prefix_un+drink[verb]+DER_V_ADJ_ABLE+PL
diff --git a/tests/germancompounding.aff b/tests/germancompounding.aff
index f1e4ea4..ed32a3b 100644
--- a/tests/germancompounding.aff
+++ b/tests/germancompounding.aff
@@ -1,7 +1,6 @@
# German compounding
-# set language for handling compound words with dash
-# and special casing of German sharp s
+# set language to handle special casing of German sharp s
LANG de_DE
@@ -49,13 +48,19 @@ SFX B 0 0/VWPXDY .
# a suffix for `Computer'
SFX C Y 2
-SFX C 0 en .
-SFX C 0 en/WXDY .
+SFX C 0 n .
+SFX C 0 n/WXDY .
-# for forbid exceptions <*Arbeitsnehmer>
+# for forbid exceptions (*Arbeitsnehmer)
FORBIDDENWORD Z
+# dash prefix for compounds with dash (Arbeits-Computer)
+
+PFX - Y 2
+PFX - 0 -/PUVW .
+PFX - 0 -/PY .
+
# decapitalizing prefix
# circumfix for positioning in compounds
diff --git a/tests/germancompounding.dic b/tests/germancompounding.dic
index c017b07..2f6d0ed 100644
--- a/tests/germancompounding.dic
+++ b/tests/germancompounding.dic
@@ -1,5 +1,5 @@
4
Arbeit/A
-Computer/BC
+Computer/BC-
-/W
Arbeitsnehmer/Z
diff --git a/tests/germancompounding.good b/tests/germancompounding.good
index 1a52f9d..d87fb04 100644
--- a/tests/germancompounding.good
+++ b/tests/germancompounding.good
@@ -1,13 +1,13 @@
Computer
-Computeren
+Computern
Arbeit
Arbeits-
Computerarbeit
Computerarbeits-
Arbeitscomputer
-Arbeitscomputeren
+Arbeitscomputern
Computerarbeitscomputer
-Computerarbeitscomputeren
+Computerarbeitscomputern
Arbeitscomputerarbeit
Computerarbeits-Computer
-Computerarbeits-Computeren
+Computerarbeits-Computern
diff --git a/tests/germancompounding.wrong b/tests/germancompounding.wrong
index a98ac43..c5f2ba1 100644
--- a/tests/germancompounding.wrong
+++ b/tests/germancompounding.wrong
@@ -1,30 +1,30 @@
computer
-computeren
+computern
arbeit
Arbeits
arbeits
ComputerArbeit
-ComputerenArbeit
-Computerenarbeit
+ComputernArbeit
+Computernarbeit
ComputerArbeits
Arbeitcomputer
-Arbeitcomputeren
+Arbeitcomputern
ArbeitsComputer
-ArbeitsComputeren
+ArbeitsComputern
Computerarbeitcomputer
ComputerArbeitcomputer
ComputerArbeitscomputer
-Computerarbeitcomputeren
-ComputerArbeitcomputeren
-ComputerArbeitscomputeren
+Computerarbeitcomputern
+ComputerArbeitcomputern
+ComputerArbeitscomputern
Arbeitscomputerarbeits
-Arbeitscomputerenarbeits
+Arbeitscomputernarbeits
Computerarbeits-computer
Arbeitsnehmer
computers
-computeren
-computerenarbeit
-computerenArbeit
+computern
+computernarbeit
+computernArbeit
computerArbeit
computerArbeits
arbeitcomputer
@@ -35,16 +35,16 @@ computerArbeitscomputer
arbeitscomputerarbeits
computerarbeits-computer
arbeitsnehmer
-computerenarbeit
-computerenArbeit
+computernarbeit
+computernArbeit
arbeits-
computerarbeit
computerarbeits-
arbeitscomputer
-arbeitscomputeren
+arbeitscomputern
computerarbeitscomputer
-computerarbeitscomputeren
+computerarbeitscomputern
computerarbeitscomputers
arbeitscomputerarbeit
computerarbeits-Computer
-computerarbeits-Computeren
+computerarbeits-Computern
diff --git a/tests/pseudoroot4.aff b/tests/pseudoroot4.aff
new file mode 100644
index 0000000..9346291
--- /dev/null
+++ b/tests/pseudoroot4.aff
@@ -0,0 +1,2 @@
+PSEUDOROOT X
+COMPOUNDFLAG Y
diff --git a/tests/pseudoroot4.dic b/tests/pseudoroot4.dic
new file mode 100644
index 0000000..96f80c1
--- /dev/null
+++ b/tests/pseudoroot4.dic
@@ -0,0 +1,5 @@
+4
+foo/X [1]
+foo/Y [2]
+foo/YX [3]
+bar/Y [4]
diff --git a/tests/pseudoroot4.good b/tests/pseudoroot4.good
new file mode 100644
index 0000000..7e4b098
--- /dev/null
+++ b/tests/pseudoroot4.good
@@ -0,0 +1,5 @@
+foo
+bar
+foobar
+barfoo
+
diff --git a/tests/pseudoroot4.test b/tests/pseudoroot4.test
new file mode 100755
index 0000000..89ca772
--- /dev/null
+++ b/tests/pseudoroot4.test
@@ -0,0 +1,4 @@
+#!/bin/sh
+DIR="`dirname $0`"
+NAME="`basename $0 .test`"
+$DIR/test_hunmorph $NAME
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/pkg-openoffice/hunspell.git
Reply to: