Task Description

 

We propose a SemEval task aimed at evaluating Word Sense Disambiguation systems in a multilingual scenario. In contrast to other complementary proposals, such as cross-lingual Word Sense Disambiguation [1] and lexical substitution [2], this task will benchmark multilingual lexical disambiguation approaches in a more “classic” setup – namely, by marking occurrences of polysemous words in different languages with sense labels provided by a multilingual sense inventory. To enable multilinguality we will make use of the sense inventory provided by BabelNet [3], a wide-coverage semantic network built by merging WordNet with Wikipedia to provide an “encyclopedic dictionary” whose concepts are lexicalized in many languages using  Wikipedia’s inter-language links and the output of a state-of-the-art machine translation system.

Following the traditional WSD “all-words” experimental setting [4], systems will be expected to link all occurrences of noun phrases within arbitrary texts in different languages to their corresponding Babel synsets (which refer to both disambiguated nouns and named entities). For instance, given the sentence:

  1. The dramatic force of Miller's play derives in part from expressionistic techniques he used to portray Loman's psychological anguish and guilt-ridden fantasy life.

a disambiguation system should link “Miller” to the Babel synset containing both the Wikipedia sense corresponding to the page http://en.wikipedia.org/wiki/Arthur_Miller, as well as Miller#n#3 (i.e. the third WordNet sense for Miller). Similarly, the occurrence of “play” in the above fragment should be linked to the Babel synset corresponding to the theatrical sense of play (as opposed to, say, its meaning as in “child’s play”):

 

{ EN:play , EN:drama, ES:obra, DE:Theaterstück, IT:dramma, FR:pièce_de_théâtre }

 

The availability of a multilingual sense inventory makes it possible to evaluate the performance of the same WSD system in different languages (i.e. other than English). For instance, given the following Italian sentence

  1. Pasolini completò inoltre il dramma in italiano in tre atti intitolato Il Cappellano e pubblicò, nelle Edizioni dell'Academiuta, la raccolta poetica, sempre in italiano, I Pianti.

the system should link the occurrence of "dramma" to the same aforementioned Babel synset referring to the theatrical sense of play.

Multilingual WSD using the sense inventory provided from an encyclopedic dictionary like BabelNet is related to the Knowledge-Base Population (KBP) Entity Linking task (a.k.a. “Wikification”), developed in the context of the recent Text Analysis Conference (TAC) campaigns. However, in contrast to the TAC Entity Linking task, what we propose here is a disambiguation task which is not restricted to named entities alone, but instead is configured as an all-words WSD task using the union of WordNet and Wikipedia as a sense inventory. In addition, in order to provide a clear picture of the disambiguation performance of the systems, the task will consist solely of a disambiguation part – i.e. no recognition of the candidate noun phrases will be required, as all NPs will be annotated and no KBP-like “NULL“ label will have to be given as system output.

 

System output


BabelNet combines WordNet and Wikipedia within a unified resource. Accordingly, Babel synsets can contain both WordNet and Wikipedia senses (i.e., pages) at the same time. We will exploit this feature for the present multilingual WSD task by allowing the participating systems to provide sense labels of three different kinds, namely any of the WordNet sense keys, Wikipedia page titles or Babel synset offsets found within the multilingual sense inventory (see the Data page for details). This way we want to encourage the participation of systems also tailored for other tasks such as 'classic' monolingual WSD or Wikification. Note, however, that the systems will be evaluated separately depending on the type of sense label they output.

 

Evaluation


We will evaluate using standard precision, recall and F1 measures used in WSD evaluation.

 

References

  1. Els Lefever and Veronique Hoste, SemEval-2010 Task 3: Cross-lingual Word Sense Disambiguation, in Proceedings of SemEval 2010, Uppsala, Sweden, 11-16 July 2010.
  2. Rada Mihalcea, Ravi Sinha and Diana McCarthy, SemEval-2010 Task 2: Cross-Lingual Lexical Substitution, in Proceedings of SemEval 2010, Uppsala, Sweden, 11-16 July 2010.
  3. Roberto Navigli and Simone Paolo Ponzetto. BabelNet: The Automatic Construction, Evaluation and Application of a Wide-Coverage Multilingual Semantic Network. Artificial Intelligence 193, 217-250.
  4. Roberto Navigli. Word Sense Disambiguation: A survey. ACM Computing Survey, 41(2), ACM Press, 2009, pp. 1-69.

Contact Info

Organizers

Roberto Navigli
 Sapienza University of Rome, Italy
David A. Jurgens
 Sapienza University of Rome, Italy

Other Info

Announcements

  • March 3, 2013: BabelNet 1.1.1 Released for Task Sense Inventory.
  • July 2012: Trial data are out! You can find them here.
  • July 2013: Test data is now publicly available. Please download the data here.