Data and Tools

 

DATA

TERMS OF USE

 

5-way Response Analysis Task Data (Training and Test Included)

 

2- and 3- way Response Analysis Task data (Training and Test included)

 

Partial Entailment Pilot Task Data (Training and Test included)

 

 

Trial data sets

 

5-way Response Analysis Task Trial Data  Contains samples from both Beetle and SciEntsBank corpora, in the format that will be used for the task. The format.html file inside the zip provides additional documentation.

 

2- and 3-way Response Analysis Task Trial Data  (UPDATED Oct. 1, 2012) Contains samples from both Beetle and SciEntsBank corpora,  in the format that will be used for the task. The format.html file inside the zip provides additional documentation.

 

NAACL 2012 data - the dataset used in (Dzikovska, Nielsen and Brew, 2012) NAACL paper. This dataset will be used as training data for the 5-way response analysis task; however, note that we are doing a second pass of data checking, which may result in a (small) number of questions being dropped, and some labels being changed. Make sure to download the final version if you are entering the shared task (to be posted by the SEMEVAL deadline).

 

TOOLS

 

 Evaluation and baseline code  (UPDATES:  Mar 10, 2013 -- bugfix to partial entailment script; Mar 4, 2013 -- bugfixes to support test-set evaluation). Evaluation scripts and baseline code for  both main task (5-way, 3-way and 2-way) and pilot task.  Includes scripts to compute all evaluation metrics, plus source code for the baseline lexical similarity classifier from the NAACL 2012 paper.

Gold Standard and Baselines for evaluation scripts - Tabular formats used to run the March 2013 challenge evaluation, includes gold standard files and baseline output files that can be given as input to evaluation scripts, both for main and pilot tasks

 

Sample file formats for evaluation scripts (UPDATE: Mar 7, 2013 - updated to match the version distributed via FTP to task participants) Includes example files to use as input to evaluation scripts distributed with the task. These are the formats that will be required for the system submissions.

 

Contact Info

Organizers


Other Info

Announcements