Feature #217

Updated by Matthieu Decorde over 3 years ago

The SimpleTokenizerXML does not use lang specifiq rules to tokenize clitics.

h3. Solution 1

Use TreeTagger clitic tokenizer rules for the fr, en and it languages as defined in the in "Gestion de la langue" section of https://groupes.renater.fr/wiki/txm-info/public/composant_de_tokenisation#solution_1_simpletokenizerxml languages.

h3. Solution 2

Use another tokenizer, to be choosen choose between existing solutions of solutions: https://groupes.renater.fr/wiki/txm-info/public/specs_import_annotation_lexicale_auto#solution