Data and Tools
Trial data sets
5-way Response Analysis Task Trial Data Contains samples from both Beetle and SciEntsBank corpora, in the format that will be used for the task. The format.html file inside the zip provides additional documentation.
2- and 3-way Response Analysis Task Trial Data (UPDATED Oct. 1, 2012) Contains samples from both Beetle and SciEntsBank corpora, in the format that will be used for the task. The format.html file inside the zip provides additional documentation.
NAACL 2012 data - the dataset used in (Dzikovska, Nielsen and Brew, 2012) NAACL paper. This dataset will be used as training data for the 5-way response analysis task; however, note that we are doing a second pass of data checking, which may result in a (small) number of questions being dropped, and some labels being changed. Make sure to download the final version if you are entering the shared task (to be posted by the SEMEVAL deadline).
Evaluation and baseline code (UPDATES: Mar 10, 2013 -- bugfix to partial entailment script; Mar 4, 2013 -- bugfixes to support test-set evaluation). Evaluation scripts and baseline code for both main task (5-way, 3-way and 2-way) and pilot task. Includes scripts to compute all evaluation metrics, plus source code for the baseline lexical similarity classifier from the NAACL 2012 paper.
Gold Standard and Baselines for evaluation scripts - Tabular formats used to run the March 2013 challenge evaluation, includes gold standard files and baseline output files that can be given as input to evaluation scripts, both for main and pilot tasks
Sample file formats for evaluation scripts (UPDATE: Mar 7, 2013 - updated to match the version distributed via FTP to task participants) Includes example files to use as input to evaluation scripts distributed with the task. These are the formats that will be required for the system submissions.