Laboratoire ICAR » Plateforme TXM

To reproduce the bug, create a source folder named "СИНДАРИН", place any XML or text file in it.
[note that the "С" and "А" characters are not the usual "C" and "A" ascii characters]
Then call any import module and select the "СИНДАРИН" folder.
TXM proposes "СИНДАРИН" for the corpus name.
Click OK.
The import process is stuck, no message appears on the console.
Close the import form, exit TXM.
Re-open TXM, set log level to "ALL" and try to start an import module.
The following message appears in the console:
"Checking corpus name validity with '[A-Z][-A-Z0-9]+': СИНДАРИН" and nothing else happens.

To restore the import capacity, you need to delete the user/TXM 0.8.0/.txm folder.

solution¶

• Do not build an invalid CQP corpus name if the folder name consists only of non ASCII characters (from the first UTF-8 plane)
• If the corpus name normalization process returns an empty sequence, display an error message and return to the corpus name definition stage
  • EN "** corpus name normalization: no valid character available to build the corpus name (try to use some American English - ASCII - characters in the folder name)"
  • FR "** normalisation du nom du corpus : aucun caractère valide disponible pour construire le nom du corpus (essayez d'utiliser des caractères anglais américain - ASCII - dans le nom du dossier)"

• Use transliteration/vocalization rules to build the normalized corpus name:

For example with International Component for Unicode for Java (ICU4J) (http://site.icu-project.org/home/why-use-icu4j, https://mvnrepository.com/artifact/com.ibm.icu/icu4j)

Groovy code:

1import com.ibm.icu.text.Transliterator
2
3cyrillic = "СИНДАРИН" 
4id = "Any-Latin; NFD; Upper; [^\\p{Alnum}] Remove" 
5latin = Transliterator.getInstance(id).transform(cyrillic)
6
7printf("translit(%s) = %s\n", cyrillic, latin)

gives:

translit(СИНДАРИН) = SINDARIN

Update AsciiUtils.converttoAscii() with the ICU tools

[adapted from https://stackoverflow.com/questions/5818912/icu4j-cyrillic-to-latin]