Laboratoire ICAR » Plateforme TXM

Manage sources with no word ids or avoid propagation errors of source ids into TXM words.

Solution 1¶

Implement parts of the strategy described here: https://groupes.renater.fr/wiki/txm-info/public/composant_de_tokenisation#strategie
(this strategy imply that a tokenizer process is always done.)

The solution creates a new import parameter : doBuildWordIDs.

In this solution, the tokenizer is activated or not (in the Tokenizer import section) :

• if the tokenizer not activated : the user must provide the word elements and set correctly the xml:id = behave like the doBuildWordId is not set
• if activated, the tokenizer can user the xml:id of pre-encoded words if the doBuildWordIds import parameter is not set. If not pre-encoded xml:id are moved to the "foreign-id" property