Laboratoire ICAR » Plateforme TXM

Well, trying to import form source without the Sorciere.xml file which is the only one containing "<pb/>" tag does not remove this issue, the same null pointer exception occurs. The second thing I notated on this file is that the "pb" tag already contains a "n" attribute which could conflict with "n" attribute added while importing ?

Even if the problem comes from an import module, the Description command should give a diagnostic message about the corpus inaccessibility - not just display nothing.

Empty pb_n index file should not clearly be a problem, as this actually happens in all corpora (and with all empty tags). Ideally, empty tags should not be indexed by the search engine to avoid confusion.
The conflict with supplied vs. automatically added @n should not be a problem either. We should investigate further.

Thanks for the precisions, I understand now.
@Serge: that's because I think the case should never appear I said the problem should be treated in import layer rather than in each command, to resolve all others issues, partition creation, internal view, etc. (but I may don't know enough the import and CQP process at this moment).

@Seb: "the case should never appear": a bug is a situation in which what should never appear appears.
My point concerns in fact our general diagnostic message policy: in a diagnostic mode (for example for a specific log level), components should not trust data coming from other components and in case of problems should diagnose what is problematic in the data from their point of view. The idea is to help the user to trace back to the original component of a problem.

@Serge: yes, thanks, you're right.

About the original issue, it seems to come from the tag "<DN_DENOM>" in the XML source files. Maybe the "_" character is the problem. Replacing "DN_DENOM" by "DNDENOM" removes the null exception on my TXM version.

Another possibility is a conflict with "<DN>" tag. The Diagnostic class or Importer may have a problem on this type of branch ("recursive naming" DN, DN_), need to prove this point:

<DN>
<DN_DENOM></DN_DENOM>
</DN>

About the Java/CQP corpus hierarchy we may try to keep a better coherent data state between Java structural unit/properties hierarchy and CQP structural unit/properties hierarchy, here is the CQP Graal log when calling Description command:

mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\front_n.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute front_n can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\hi_rend.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute hi_rend can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\hi_n.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute hi_n can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\lb_ed.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute lb_ed can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\lb_type.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute lb_type can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\lb_n.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute lb_n can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\milestone_n.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute milestone_n can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\milestone_ed.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute milestone_ed can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\milestone_unit.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute milestone_unit can't be loaded
mmapfile()<storage.c>: Can't mmap() file C:\Users\s\TXM\corpora\graal\data\GRAAL\pb_n.rng ...
    You have probably run out of memory / address space!
    Error Message: attributes:load_component(): Warning:
  Data of STRUC component of attribute pb_n can't be loaded

plus, it leads to subcorpus or partition possibilities with no tokens.

'it seems to come from the tag "<DN_DENOM>" in the XML source files'

XML sources tag names MUST not contain '_' characters because the CQP indexing system uses it to manage pseudo-recursion of XML elements (recursion is flatten by creating new element names by concatenating recursion informations separated by '_').

• file paths (no accents or space, etc?)
• file names (.xml extension?)
• xml element names (no underscore?)
• xml attribute names
• xml entities constraints (no entities?)

Then we must enforce that policy BEFORE CQP indexing process because CQP indexing doesn't enforce a strict policy with respect to its own constraints.

Then we must document that policy in the TXM manual.

Some other remarks :

1) I can get Description on the 3 corpora made for each text separately. In particular, it works for corpus Perrault7 which contains <DN_DENOM> tags.

• Échec de récupération de la dimension du corpus CORPUSCONTES
  [...]

• a) it may be possible to use the '_' character in tag names in XML sources (difficult to understand the impact à CQL queries expressing contraints on structural property values and using the special '_' character - eg [_.text_loc="dg"])
• b) it seems impossible to use the '.' character in tag names in XML sources (the error message is related the closing tag but we can try to extrapolate. Same remark about expressing constraints on structural property values in CQL queries because the '.' character has a special role there - eg [_.text_loc="dg"])

Conclusion: I once sent an email summarizing what I understood of the constraints related to tag names in CQP source code. We should find that, put it in a spec and enforce an input policy as detailed above in the TXM XML import modules.

This bug seems to have been fixed with last commit by testing if the value is null in Diagnostic.htmldump().
For information, with the original unfixed code:
- I have the same behavior than Bénédicte, each text imported separately works but not together.
- here is the description which shows a possible conflict about the <DE_DENOM> tag, maybe because of the "_" character or the hierarchy <DN><DN_DENOM>.

• dn
◦ denom (1) = "".
◦ n()...
◦ n1()...
• dn_denom
◦ n (6) = 0, 1, 2, 3, 4, 5.

(There are no <DENOM> tags in source files, only <DN_DENOM> tags)

Other notes on current import process that we may write down somewhere:

• the "_" and "." characters in an XML tag name seem valid if I understand well the W3C recommendations: http://www.w3.org/TR/xml/#sec-common-syn (a few tricky to read)
• when "." character is present in a source tag name, the structure seems to be ignored while importing in TXM, e.g. here: "TEI.2" tag is simply removed

Sebastien Jacquot a écrit :

• dn
◦ denom (1) = "".
◦ n()...
◦ n1()...
• dn_denom
◦ n (6) = 0, 1, 2, 3, 4, 5.

(There are no <DENOM> tags in source files, only <DN_DENOM> tags)

Actually this behavior is still present in last SVN. The internal structure hierarchy stores a dn => denom which seems an error because the denom tag is not present in corpora leading to some wrong possibilities, e.g. partition or subcorpus on DN => DENOM.