Results

System results for Task 12. Systems are group according to which sense inventory they used: BabelNet, Wikipedia, or WordNet.

The First Sense baselines were created by selecting the most frequent sense from SemCor for the WordNet and BabelNet sense inventories.  For Wikipedia, the first sense baseline was created by selecting the first Wikipedia page associated with a WordNet sense order according to SemCor frequency.

 

BabelNet systems' results

System Language Precision Recall F-score
DAEBAK DE 0.592 0.591 0.591
  EN 0.605 0.604 0.604
  ES 0.600 0.600 0.6
  FR 0.539 0.538 0.538
  IT 0.614 0.613 0.613
GETALP-BN-1 DE 0.404 0.404 0.404
  EN 0.263 0.263 0.263
  ES      
  FR 0.262 0.261 0.261
  IT 0.324 0.324 0.324
GETALP-BN-2 DE 0.401 0.400 0.400
  EN 0.266 0.266 0.266
  ES 0.371 0.371 0.371
  FR 0.258 0.257 0.257
  IT 0.325 0.325 0.325
MFS DE 0.676 0.673 0.686
  EN 0.665 0.665 0.656
  ES 0.645 0.645 0.644
  FR 0.455 0.452 0.501
  IT 0.576 0.574 0.572
UMCC-DLSI-1 DE 0.619 0.617 0.618
  EN 0.677 0.677 0.677
  ES 0.708 0.703 0.705
  FR 0.608 0.603 0.605
  IT 0.659 0.656 0.657
UMCC-DLSI-2 DE 0.622 0.620 0.621
  EN 0.685 0.685 0.685
  ES 0.713 0.708 0.710
  FR 0.608 0.603 0.605
  IT 0.659 0.657 0.658
UMCC-DLSI-3 EN 0.680 0.680 0.68

Wikipedia systems' results

System Language Precision Recall F-score
MFS DE 0.836 0.827 0.831
  EN 0.860 0.753 0.803
  ES 0.830 0.819 0.824
  FR 0.698 0.691 0.694
  IT 0.833 0.813 0.823
UMCC-DLSI-1 DE 0.758 0.460 0.572
  EN 0.619 0.484 0.543
  ES 0.773 0.493 0.602
  FR 0.817 0.480 0.605
  IT 0.785 0.458 0.578
UMCC-DLSI-2 DE 0.769 0.467 0.581
  EN 0.620 0.487 0.546
  ES 0.778 0.502 0.610
  FR 0.815 0.478 0.603
  IT 0.787 0.463 0.583
UMCC-DLSI-3 EN 0.622 0.489 0.548

WordNet systems' results

System Language Precision Recall F-score
GETALP-WN-1 EN 0.406 0.406 0.406
MFS EN 0.630 0.630 0.630
UMCC-DLSI-3 EN 0.642 0.639 0.640
UMCC-DLSI-1 EN 0.639 0.635 0.637
UMCC-DLSI-2 EN 0.649 0.645 0.647

 

Contact Info

Organizers

Roberto Navigli
 Sapienza University of Rome, Italy
David A. Jurgens
 Sapienza University of Rome, Italy

Other Info

Announcements

  • March 3, 2013: BabelNet 1.1.1 Released for Task Sense Inventory.
  • July 2012: Trial data are out! You can find them here.
  • July 2013: Test data is now publicly available. Please download the data here.