Bug #1313
Mis à jour par Serge Heiden il y a presque 10 ans
It is impossible to import text containing Unicode characters in some ranges.
The following ranges have been reported to cause an error and interrup the import process:
* Cuneiform characters range (Akkadian corpus)
* Smiley range (SMS corpus)
* Classical Chinese characters range (Taiwan corpus)
h3. Diagnostic
It has been observed that for Unicode characters outside specific ranges, the XML libraries used by TXM replace the native character code by an XML numerical character entity of the form '@&<character code point>;@'.
Further import processing steps seems not able to process XML numerical character entities, so the import process is interrupted.
h3. How to reproduce the error
h4. Cuneiform characters range (Akkadian corpus)
The following error occurs when validating a tokenized text (it looks like the characters are incorrectly handles by the tokenizer and invalid XML characters are produced):
<code>
<pre>
Execution du script : /home/alavrent/TXM/scripts/import/xmlLoader.groovy
Trying to read import properties file: /home/alavrent/xml/corpusakkadiencuneiform/import.properties
Trying to read metadatas from: /home/alavrent/xml/corpusakkadiencuneiform/metadata.csv
no metadata file: /home/alavrent/xml/corpusakkadiencuneiform/metadata.csv
-- IMPORTER - Reading source files
Sources clean & validation
.
Files processed: [/home/alavrent/TXM/corpora/corpusakkadiencuneiform/txm/CORPUSAKKADIENCUNEIFORM/AS_22_6_cuneif-unicode.xml]
Tokenizing 1 files
.
Building XML-TXM (1 files)
.Unexpected error while parsing file file:/home/alavrent/TXM/corpora/corpusakkadiencuneiform/tokenized/AS_22_6_cuneif-unicode.xml : javax.xml.stream.XMLStreamException: ParseError at [row,col]:[9,9]
Message: La référence de caractère "&#
Location line: 9 character: 9
Failed to process /home/alavrent/TXM/corpora/corpusakkadiencuneiform/tokenized/AS_22_6_cuneif-unicode.xml
</code>
To reproduce the bug, use the /SpUV/Corpus Akkadien - M. Beranger/Marine_avril_2015-2/AS_22_6_cuneif-unicode.xml file for XML/W+CSV import
</pre>
h4. Smiley range (SMS corpus)
Importing 88milSMS corpus (http://88milsms.huma-num.fr/corpus.html).
- *TXT+CSV import* module (June 2015) applied to UTF-8 TXT:
<pre>
J'irai me renseigner aussi ( si j'trouve le batiment
The following ranges have been reported to cause an error and interrup the import process:
* Cuneiform characters range (Akkadian corpus)
* Smiley range (SMS corpus)
* Classical Chinese characters range (Taiwan corpus)
h3. Diagnostic
It has been observed that for Unicode characters outside specific ranges, the XML libraries used by TXM replace the native character code by an XML numerical character entity of the form '@&<character code point>;@'.
Further import processing steps seems not able to process XML numerical character entities, so the import process is interrupted.
h3. How to reproduce the error
h4. Cuneiform characters range (Akkadian corpus)
The following error occurs when validating a tokenized text (it looks like the characters are incorrectly handles by the tokenizer and invalid XML characters are produced):
<code>
<pre>
Execution du script : /home/alavrent/TXM/scripts/import/xmlLoader.groovy
Trying to read import properties file: /home/alavrent/xml/corpusakkadiencuneiform/import.properties
Trying to read metadatas from: /home/alavrent/xml/corpusakkadiencuneiform/metadata.csv
no metadata file: /home/alavrent/xml/corpusakkadiencuneiform/metadata.csv
-- IMPORTER - Reading source files
Sources clean & validation
.
Files processed: [/home/alavrent/TXM/corpora/corpusakkadiencuneiform/txm/CORPUSAKKADIENCUNEIFORM/AS_22_6_cuneif-unicode.xml]
Tokenizing 1 files
.
Building XML-TXM (1 files)
.Unexpected error while parsing file file:/home/alavrent/TXM/corpora/corpusakkadiencuneiform/tokenized/AS_22_6_cuneif-unicode.xml : javax.xml.stream.XMLStreamException: ParseError at [row,col]:[9,9]
Message: La référence de caractère "&#
Location line: 9 character: 9
Failed to process /home/alavrent/TXM/corpora/corpusakkadiencuneiform/tokenized/AS_22_6_cuneif-unicode.xml
</code>
To reproduce the bug, use the /SpUV/Corpus Akkadien - M. Beranger/Marine_avril_2015-2/AS_22_6_cuneif-unicode.xml file for XML/W+CSV import
</pre>
h4. Smiley range (SMS corpus)
Importing 88milSMS corpus (http://88milsms.huma-num.fr/corpus.html).
- *TXT+CSV import* module (June 2015) applied to UTF-8 TXT:
<pre>
J'irai me renseigner aussi ( si j'trouve le batiment