Feature #217
Mis à jour par Matthieu Decorde il y a plus de 8 ans
The SimpleTokenizerXML does not use lang specifiq rules to tokenize clitics.
h3. Solution 1
Use TreeTagger clitic tokenizer rules for the fr, en and it languages as defined in the in "Gestion de la langue" section of https://groupes.renater.fr/wiki/txm-info/public/composant_de_tokenisation#solution_1_simpletokenizerxml languages.
h3. Solution 2
Use another tokenizer, to be choosen choose between existing solutions of solutions: https://groupes.renater.fr/wiki/txm-info/public/specs_import_annotation_lexicale_auto#solution
h3. Solution 1
Use TreeTagger clitic tokenizer rules for the fr, en and it languages as defined in the in "Gestion de la langue" section of https://groupes.renater.fr/wiki/txm-info/public/composant_de_tokenisation#solution_1_simpletokenizerxml languages.
h3. Solution 2
Use another tokenizer, to be choosen choose between existing solutions of solutions: https://groupes.renater.fr/wiki/txm-info/public/specs_import_annotation_lexicale_auto#solution