Task Description

Abstract:

We propose a second edition of the "Cross-Lingual Word Sense Disambiguation" task, that builds further on the insights we gained from the SemEval-2010 evaluation and for which new test data will be annotated.  The task is an unsupervised Word Sense Disambiguation task for English nouns by means of parallel corpora. The sense label is composed of translations in the different target languages (viz. French, Italian, Spanish, Dutch and German). The sense inventory is built up on the basis of the Europarl parallel corpus; all translations of a polysemous word are grouped into clusters, which constitute different senses of that given word. For the test data, native speakers decide on the correct translation cluster(s) for each test sentence and give their top three translations from the predefined list of Europarl translations, in order to assign weights to the set of gold standard translations. Systems can participate in both a bilingual evaluation (where translations in one target language are evaluated) or a multilingual evaluation (where translations in all five target languages are evaluated). The evaluation will be done using precision and recall measures and we will perform both a "best result" evaluation (the first translation returned by a system) and a more relaxed evaluation for the "top five" results (the first five translations returned by a system).


INTRODUCTION

There is a general feeling in the WSD community that WSD should not be considered as an isolated research task, but should be integrated in real NLP applications such as Machine translation or multilingual IR. Using translations from a corpus instead of human defined (e.g. WordNet) sense labels, makes it easier to integrate WSD in multilingual applications, and solves the granularity problem that might be task-dependent as well. Furthermore, this type of corpus-based approach is language-independent and can be a valid alternative for languages that lack sufficient sense-inventories and sense-tagged corpora.
We organised a first "Cross-lingual Word Sense Disambiguation" task at SemEval-2010 (E.Lefever and V.Hoste, 2010) for which we received 16 submissions from five different research teams. Many additional research teams showed their interest and downloaded the trial data, but did not manage to finish their systems in time.
In order to gain more insights into the complexity and the viability of cross-lingual WSD, we propose a second edition of the task for which new test data will be annotated.


TASK DESCRIPTION

We propose an Unsupervised Word Sense Disambiguation task for English nouns by means of parallel corpora. The sense label is composed of translations in the different languages and the sense inventory is built up on the basis of the Europarl parallel corpus (http://www.statmt.org/europarl/). All translations of a polysemous word are grouped into clusters/"senses" of that given word. The sense inventory for all target nouns was manually built up on the basis of all retrieved translations from Europarl.

Languages: English - Dutch, French, German, Italian, Spanish

Two Subtasks:

1. Bilingual Evaluation (English - Language X)

Example:
[English] ... equivalent to giving fish to people living on the bank of the river ...
[Dutch] ... dit komt erop neer dat dorpelingen aan de oever van de rivier vis wordt gegeven ...

Sense Label = {oever/dijk} [Dutch]
Sense Label = {rives/rivage/bord/bords} [French]
Sense Label = {Ufer} [German]
Sense Label = {riva} [Italian]
Sense Label = {orilla} [Spanish]

Example:
[English] The [bank] of Scotland

Sense Label = {bank/kredietinstelling} [Dutch]
Sense Label = {banque/établissement de crédit} [French]
Sense Label = {Bank/Kreditinstitut} [German]
Sense Label = {banca} [Italian]
Sense Label = {banco} [Spanish]


2. Multi-lingual Evaluation (English - all target languages)

Example:
The river [bank]
Sense Label = {oever/dijk, rives/rivage/bord/bords, Ufer, riva, orilla}

The [bank] of Scotland
Sense Label = {bank/kredietinstelling, banque/établissement de crédit, Bank/Kreditinstitut, banca, banco}


RESOURCES

As the task is formulated as an unsupervised WSD task, we will not annotate any training material. Participants can use the Europarl corpus that is freely available and that was used for building up the sense inventory. Participants are free to use other training corpora, but additional senses/translations (which are not present in Europarl) will not be included in the sense inventory that is used for evaluation.

The development data consists of the trial and test data of the previous SemEval Cross-lingual Word Sense Disambiguation task.

We will manually annotate new test data:
for the test data, native speakers will decide on the correct translation cluster(s) for each test sentence and give their top-3 translations from the predefined list of Europarl translations (see EVALUATION).

- Development/sample data:
- give a preview of how the gold standard test data will look like (ambiguous nouns get sense label)
- trial and test data from the previous SemEval competition (25 ambiguous words and a total of 1100 labeled instances)

- Test data:
- indicate the ambiguous nouns that should be tagged
- we will stick to the same set of 20 ambiguous nouns from the previous SemEval competition
- total test set of 20 polysemous nouns x 50 instances per word

EVALUATION

We will use an evaluation scheme that is inspired by the English lexical substitution task in SemEval 2010.
The evaluation will be done using precision and recall. We will perform both a "best result" evaluation (the first translation returned by a system) and a more relaxed evaluation for the "top five" results (the first five translations returned by a system).

In order to assign weights to the candidate translations in the answer cluster(s) for each test sentence, native speakers will pick the three most appropriate translations from the predefined sense inventory, and these translations get higher weights (+3 if chosen by all three natives, +2 if chosen by two natives, +1 if chosen by one native).

EFFORT

Three native speakers will assign the correct translation cluster(s) to each test sentence, and give their top-3 translations for each ambiguous noun (chosen from the predefined translation list based on Europarl).

REFERENCES

Els Lefever and Véronique Hoste (2010): "SemEval-2010 Task 3: Cross-Lingual Word Sense Disambiguation", Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 15-20, Uppsala, Sweden, 15-16 July 2010.

Contact Info

Organizers


Els Lefever and Véronique Hoste, Ghent University College / Ghent University, Belgium

email : els.lefever@hogent.be

Other Info

Announcements