Bug #1592

Updated by Matthieu Decorde almost 4 years ago

2 main bugs has been found :

"Bar-le-Duc" is tokenized : "Bar" "-l"
"-Duc" is dropped and "-le" is truncated

Qu'est-ce is tokenized "Qu'" "-ce"
"est" is dropped

"mont-d'or" is tokenized "mont-d'" "or" instead of "mont-d'or"

h3. Solution

There was an error in the french clitic regular expression and the neighborhood characters of the clitics was not fully processed (no iteration).

There was an error in the elision regular expression the "X'" elision must be at the begining of the token.

Use the TreeTagger pclitic tokenization rules and replace the "'" quote with multiple quotes regular expression (We can't use the unicode category since the quote category is 'Po' and it contains other word separating characters : http://www.fileformat.info/info/unicode/category/Po/list.htm)

h3. Validation test

Import with clipboard in FR with the following content:
<pre>
le mot Bar-le-Duc
le mot mont-d'or
le mot est-il ?
c'est comme-ci comme-là
mot-composé-de-tirets
l'apostrophe ne pose pas d'problème !
mot composé assemblée_générale
c'est "la fin" pour aujourd'hui.
</pre>

The internal view should be composed of 3 pages:
<pre>
le
mot
Bar-le-Duc
le
mot
mont-d'or
le
mot
est
-il
?
</pre>
<pre>
c'
est
comme
-ci
comme
-là
mot-composé-de-tirets
l'
apostrophe
ne
pose
pas
d'
problème
!
</pre>
<pre>
mot
composé
assemblée_générale
c'
est
"
la
fin
"
pour
aujourd'hui
.
</pre>

Back