Bug #2709

Updated by Matthieu Decorde 6 months ago

To reproduce the bug, create a source folder named "СИНДАРИН", place any XML or text file in it.
[note that the "С" and "А" characters are not the usual "C" and "A" ascii characters]
Then call any import module and select the "СИНДАРИН" folder.
TXM proposes "СИНДАРИН" for the corpus name.
Click OK.
The import process is stuck, no message appears on the console.
Close the import form, exit TXM.
Re-open TXM, set log level to "ALL" and try to start an import module.
The following message appears in the console:
"Checking corpus name validity with '[A-Z][-A-Z0-9]+': СИНДАРИН" and nothing else happens.

To restore the import capacity, you need to delete the user/TXM 0.8.0/.txm folder.

h3. solution

Possible solutions:
* Do not build an invalid CQP corpus name if the folder name consists only of non ASCII characters (from the first UTF-8 plane)
* If the corpus name normalization process returns an empty sequence, display an error message and return to the corpus name definition stage
** EN
"** corpus name normalization: no valid character available to build the corpus name (try to use some American English - ASCII - characters in the folder name)"
** FR ""

and return to the corpus name definition stage

Possible help for the user:
* Use transliteration/vocalization rules to build the normalized corpus name:

For example with International Component for Unicode for Java (ICU4J) (http://site.icu-project.org/home/why-use-icu4j, https://mvnrepository.com/artifact/com.ibm.icu/icu4j)

Groovy code:
<pre><code class="groovy">
import com.ibm.icu.text.Transliterator

cyrillic = "СИНДАРИН"
id = "Any-Latin; NFD; Upper; [^\\p{Alnum}] Remove"
latin = Transliterator.getInstance(id).transform(cyrillic)

printf("translit(%s) = %s\n", cyrillic, latin)
</code></pre>

gives:

<pre>
translit(СИНДАРИН) = SINDARIN
</code></pre>

Update AsciiUtils.converttoAscii() with the ICU tools


[adapted from https://stackoverflow.com/questions/5818912/icu4j-cyrillic-to-latin]

Back