The task is Chinese lexical sample as well as all-words task.
1. All-words task, the test data consists of 669 sentences of running text from the People's Daily news.
2. Lexical sample task, we select 99 lemmas (27 verbs, 30 nouns, 12 adjectives and 30 adverbs). To each lemma, we will tag total about 100*(the sense numbers of that lemma) instances. The tagging data are splited into 2 parts. 2/3 will be used as training data and the rest 1/3 is test data. All the instances are extracted from Dynamic Circulating Corpus (DCC) developed by Chinese National Language Resources Monitoring and Research Center (CNLR).
The task includes 2 sub-tasks and one umbrella sub-task:
Sub-task#1: All-words task
Sub-task#2: Lexical sample task
Sub-task#3: the usage of Chinese adverb sample task
This is the first Chinese all-words task. The procedure of evaluation of all-words and the lexical sample task are similar to the evaluation tasks on the previous Semeval and Senseval evaluation.
The usage of Chinese adverb sample task is a new task. Adverbs are one of the main word categories. Previous research mainly focused on nouns and verbs. Adverb usage describes how the adverb is used in real text, emphasizing the syntactic and semantic combination of the adverb and its context. The granularity of adverb usage is finer than its sense. The same sense of an adverb may correspond to several different usages. So we added a Special sub-task: The usage of Chinese adverb sample task.
The sense entries of sub task #1 and #2 are specified in the Hownet (http://www.keenage.com).
The usages of Chinese adverb entries are specified in the Chinese Function Word Usages' Knowledge Base (CFKB) developed by Zhengzhou University.
We provide all the lexcials appeared in the task.
PREPARING THE TRAINING/TESTING DATA
Sub-task #1. In every sentence of the dataset, for all verbs, nouns, adjectives which have more than 1 "DEF" in the Hownet aree double blind-annotated with Hownet and then corrected by a third person.
Sub-task #2. To each target lemma(noun, verb, adjective), we select out about total 100*(the sense numbers of that lemma) instances for each word from Dynamic Circulating Corpus (DCC) developed by Chinese national language resources monitoring and research center (CNLR) randomly. DCC has 10 years (2002-2011) data of Chinese selected newspapers till now and the data of each year has about 500,000,000 tokens. All of the sentences of these two tasks have been word-segmented and part of speech (POS) tagged by CNLR. To all instances in the sample task, the target words are double blind-annotated with Hownet and then corrected by a third person. The tagging data will be split into 2 parts. 2/3 will be used as training data and the rest 1/3 is test data.
Sub-task #3. To each target lemma(adverb), we select out about total 100*(the usage numbers of that lemma) instances for each word from DCC. To all instances in the sample task, the target words are double blind-annotated with CFKB and then corrected by a third person. The tagging data will be split into 2 parts. 2/3 will be used as training data and the rest 1/3 is test data.
For All-words task, precision and recall will be used.
For the rest tasks, we will use the micro-average and macro-average similar to the Semeval-2007 task #5.
To sub-task #2, we also will make an attempt of coarse-grained-like evaluation. Here, the "DEF" labeled by different annotators on the same lemma(During our label procedure we selected this data intendedly) will be regards as one sense group.
Peng Jin, Yunfang Wu and Shiwen Yu. 2007. SemEval-2007 Task 05: Multilingual Chinese-English Lexical Sample. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), 2007:19-23