Bug #2709

RCP: 0.8.0, Import is stuck if the corpus name consists only of cyrillic characters

Added by Alexey Lavrentev 6 months ago. Updated about 1 month ago.

Status:New Start date:01/08/2020
Priority:High Due date:
Assignee:- % Done:

80%

Category:Import Spent time: -
Target version:TXM 0.8.1

Description

To reproduce the bug, create a source folder named "СИНДАРИН", place any XML or text file in it.
[note that the "С" and "А" characters are not the usual "C" and "A" ascii characters]
Then call any import module and select the "СИНДАРИН" folder.
TXM proposes "СИНДАРИН" for the corpus name.
Click OK.
The import process is stuck, no message appears on the console.
Close the import form, exit TXM.
Re-open TXM, set log level to "ALL" and try to start an import module.
The following message appears in the console:
"Checking corpus name validity with '[A-Z][-A-Z0-9]+': СИНДАРИН" and nothing else happens.

To restore the import capacity, you need to delete the user/TXM 0.8.0/.txm folder.

solution

  • Do not build an invalid CQP corpus name if the folder name consists only of non ASCII characters (from the first UTF-8 plane)
  • If the corpus name normalization process returns an empty sequence, display an error message and return to the corpus name definition stage
    • EN "** corpus name normalization: no valid character available to build the corpus name (try to use some American English - ASCII - characters in the folder name)"
    • FR "** normalisation du nom du corpus : aucun caractère valide disponible pour construire le nom du corpus (essayez d'utiliser des caractères anglais américain - ASCII - dans le nom du dossier)"
Possible help for the user:
  • Use transliteration/vocalization rules to build the normalized corpus name:

For example with International Component for Unicode for Java (ICU4J) (http://site.icu-project.org/home/why-use-icu4j, https://mvnrepository.com/artifact/com.ibm.icu/icu4j)

Groovy code:

1import com.ibm.icu.text.Transliterator
2
3cyrillic = "СИНДАРИН" 
4id = "Any-Latin; NFD; Upper; [^\\p{Alnum}] Remove" 
5latin = Transliterator.getInstance(id).transform(cyrillic)
6
7printf("translit(%s) = %s\n", cyrillic, latin)

gives:

translit(СИНДАРИН) = SINDARIN

Update AsciiUtils.converttoAscii() with the ICU tools

[adapted from https://stackoverflow.com/questions/5818912/icu4j-cyrillic-to-latin]

Associated revisions

Revision 2495
Added by Matthieu Decorde 6 months ago

fix AscciUtils.convertToAscii using ICU tools refs #2709

History

#1 Updated by Serge Heiden 6 months ago

  • Subject changed from RCP: 0.8.0, Import is stuck if the corpus name consists of cyrillic characters to RCP: 0.8.0, Import is stuck if the corpus name consists only of cyrillic characters
  • Description updated (diff)

#2 Updated by Serge Heiden 6 months ago

  • Description updated (diff)
  • Priority changed from Normal to High

#3 Updated by Matthieu Decorde 6 months ago

  • Description updated (diff)
  • % Done changed from 0 to 70

#4 Updated by Serge Heiden 6 months ago

  • Description updated (diff)

#5 Updated by Sebastien Jacquot 6 months ago

  • Target version changed from TXM 0.8.2 to TXM 0.8.1

#6 Updated by Matthieu Decorde about 1 month ago

  • % Done changed from 70 to 80

Also available in: Atom PDF