Laboratoire ICAR » Plateforme TXM

- import corpus A with XML/w module with 'en' tokenization and TreeTagger annotation
- corpus A is OK
- import corpus B with TXT+CSV with 'fr' tokenization and TreeTagger annotation
- corpus B has an english tokenization instead of french tokenization

Corpus A file.xml

This corpus works fine. That's all.

Corpus B file.txt

C'est une phrase qui ne s'tokenize pas bien.

Solution 1¶

MD: Strings with a punctuation followed by a clitics are not well tokenized

Fix the tokenizer punctuation regular expression to process the punctuation before processing the clitics punctuation.

Solution 2¶

The problem was not caused by the language parameter but from the rule to tokenize clitics.

the corpus content was more like

C' est une phrase qui ne s' tokenize pas bien.

and the tokenizer rule was "${cliticrule}.+"

The solution is to change the rule to "${cliticrule}.*"