(April 10th, bug in Mean and RankMean fixed on Apr. 12)

These are the results of the submitted systems in alphabetic order. The columns stand for:

  • ALL: Pearson correlation with the gold standard for the five datasets, and corresponding rank.
  • ALLnrm: Pearson correlation after the system outputs for each dataset are fitted to the gold standard using least squares, and corresponding rank.
  • Mean: Weighted mean across the 5 datasets, where the weight depends on the number of pairs in the dataset.
  • Followed by Pearson for individual datasets.

Please check the results using the gold standard available for download (and the evaluation script provided with the train), and let us know if there are any issues.

Changes with respect to the previous version are mentioned in the bottom.

Run ALL Rank ALLnrm RankNrm Mean RankMean MSRpar MSRvid SMT-eur On-WN SMT-news
00-baseline/task6-baseline ,3110 87 ,6732 85 ,4356 70 ,4334 ,2996 ,4542 ,5864 ,3908
aca08ls/task6-University_Of_Sheffield-Hybrid ,6485 34 ,8238 15 ,6100 18 ,5166 ,8187 ,4859 ,6676 ,4280
aca08ls/task6-University_Of_Sheffield-Machine_Learning ,7241 17 ,8169 18 ,5750 38 ,5166 ,8187 ,4859 ,6390 ,2089
aca08ls/task6-University_Of_Sheffield-Vector_Space ,6054 48 ,7946 44 ,5943 27 ,5460 ,7241 ,4858 ,6676 ,4280
acaputo/task6-UNIBA-DEPRI ,6141 46 ,8027 38 ,5891 31 ,4542 ,7673 ,5126 ,6593 ,4636
acaputo/task6-UNIBA-LSARI ,6221 44 ,8079 30 ,5728 40 ,3886 ,7908 ,4679 ,6826 ,4238
acaputo/task6-UNIBA-RI ,6285 41 ,7951 43 ,5651 45 ,4128 ,7612 ,4531 ,6306 ,4887
baer/task6-UKP-run1 ,8117 4 ,8559 4 ,6708 4 ,6821 ,8708 ,5118 ,6649 ,4672
baer/task6-UKP-run2_plus_postprocessing_smt_twsi ,8239 1 ,8579 2 ,6773 1 ,6830 ,8739 ,5280 ,6641 ,4937
baer/task6-UKP-run3_plus_random ,7790 8 ,8166 19 ,4320 71 ,6830 ,8739 ,5280 -,0620 -,0520
croce/task6-UNITOR-1_REGRESSION_BEST_FEATURES ,7474 13 ,8292 12 ,6316 10 ,5695 ,8217 ,5168 ,6591 ,4713
croce/task6-UNITOR-2_REGRESSION_ALL_FEATURES ,7475 12 ,8297 11 ,6323 9 ,5763 ,8217 ,5102 ,6591 ,4713
croce/task6-UNITOR-3_REGRESSION_ALL_FEATURES_ALL_DOMAINS ,6289 40 ,8150 21 ,5939 28 ,4686 ,8027 ,4574 ,6591 ,4713
csjxu/task6-PolyUCOMP-RUN1 ,6528 31 ,7642 59 ,5492 51 ,4728 ,6593 ,4835 ,6196 ,4290
danielcer/stanford_fsa ,6354 38 ,7212 70 ,4848 66 ,3795 ,5350 ,4377 ,6052 ,4164
danielcer/stanford_pdaAll ,4229 77 ,7160 72 ,5044 62 ,4409 ,4698 ,4558 ,6468 ,4769
danielcer/stanford_rte ,5589 55 ,7807 55 ,4674 67 ,4374 ,8037 ,3533 ,3077 ,3235
davide_buscaldi/task6-IRIT-pg1 ,4280 76 ,7379 65 ,5009 63 ,4295 ,6125 ,4952 ,5387 ,3614
davide_buscaldi/task6-IRIT-pg3 ,4813 68 ,7569 61 ,5202 58 ,4171 ,6728 ,5179 ,5526 ,3693
davide_buscaldi/task6-IRIT-wu ,4064 81 ,7287 69 ,4898 65 ,4326 ,5833 ,4856 ,5317 ,3480
demetrios_glinos/task6-ATA-BASE ,3454 83 ,6990 81 ,2772 87 ,1684 ,6256 ,2244 ,1648 ,0988
demetrios_glinos/task6-ATA-CHNK ,4976 64 ,7160 73 ,3215 86 ,2312 ,6595 ,1504 ,2735 ,1426
demetrios_glinos/task6-ATA-STAT ,4165 79 ,7129 75 ,3312 85 ,1887 ,6482 ,2769 ,2950 ,1336
desouza/task6-FBK-run1 ,5633 54 ,7127 76 ,3628 82 ,2494 ,6117 ,1495 ,4212 ,2439
desouza/task6-FBK-run2 ,6438 35 ,8080 29 ,5888 32 ,5128 ,7807 ,3796 ,6228 ,5474
desouza/task6-FBK-run3 ,6517 32 ,8106 25 ,6077 20 ,5169 ,7773 ,4419 ,6298 ,6085
dvilarinoayala/task6-BUAP-RUN-1 ,4997 63 ,7568 62 ,5260 56 ,4037 ,6532 ,4521 ,6050 ,4537
dvilarinoayala/task6-BUAP-RUN-2 -,0260 89 ,5933 89 ,1016 89 ,1109 ,0057 ,0348 ,1788 ,1964
dvilarinoayala/task6-BUAP-RUN-3 ,6630 25 ,7474 64 ,5105 59 ,4018 ,6378 ,4758 ,5691 ,4057
enrique/task6-UNED-H34measures ,4381 75 ,7518 63 ,5577 48 ,5328 ,5788 ,4785 ,6692 ,4465
enrique/task6-UNED-HallMeasures ,2791 88 ,6694 87 ,4286 72 ,3861 ,2570 ,4086 ,6006 ,5305
enrique/task6-UNED-SP_INIST ,4680 69 ,7625 60 ,5615 47 ,5166 ,6303 ,4625 ,6442 ,4753
georgiana_dinu/task6-SAARLAND-ALIGN_VSSIM ,4952 65 ,7871 50 ,5065 60 ,4043 ,7718 ,2686 ,5721 ,3505
georgiana_dinu/task6-SAARLAND-MIXT_VSSIM ,4548 71 ,8258 13 ,5662 43 ,6310 ,8312 ,1391 ,5966 ,3806
jan_snajder/task6-takelab-simple ,8133 3 ,8635 1 ,6753 2 ,7343 ,8803 ,4771 ,6797 ,3989
jan_snajder/task6-takelab-syntax ,8138 2 ,8569 3 ,6601 5 ,6985 ,8620 ,3612 ,7049 ,4683
janardhan/task6-janardhan-UNL_matching ,3431 84 ,6878 84 ,3481 83 ,1936 ,5504 ,3755 ,2888 ,3387
jhasneha/task6-Penn-ELReg ,6622 27 ,8048 34 ,5654 44 ,5480 ,7844 ,3513 ,6040 ,3607
jhasneha/task6-Penn-ERReg ,6573 28 ,8083 28 ,5755 37 ,5610 ,7857 ,3568 ,6214 ,3732
jhasneha/task6-Penn-LReg ,6497 33 ,8043 36 ,5699 41 ,5460 ,7818 ,3547 ,5969 ,4137
jotacastillo/task6-SAGAN-RUN1 ,5522 57 ,7904 47 ,5906 29 ,5659 ,7113 ,4739 ,6542 ,4253
jotacastillo/task6-SAGAN-RUN2 ,6272 42 ,8032 37 ,5838 34 ,5538 ,7706 ,4480 ,6135 ,3894
jotacastillo/task6-SAGAN-RUN3 ,6311 39 ,7943 45 ,5649 46 ,5394 ,7560 ,4181 ,5904 ,3746
Konstantin_Z/task6-ABBYY-General ,5636 53 ,8052 33 ,5759 36 ,4797 ,7821 ,4576 ,6488 ,3682
M_Rios/task6-UOW-LEX_PARA ,6397 36 ,7187 71 ,3825 80 ,3628 ,6426 ,3074 ,2806 ,2082
M_Rios/task6-UOW-LEX_PARA_SEM ,5981 49 ,6955 82 ,3473 84 ,3529 ,5724 ,3066 ,2643 ,1164
M_Rios/task6-UOW-SEM ,5361 59 ,6287 88 ,2567 88 ,2995 ,2910 ,1611 ,2571 ,2212
mheilman/task6-ETS-PERP ,7808 7 ,8064 32 ,6305 11 ,6211 ,7210 ,4722 ,7080 ,5149
mheilman/task6-ETS-PERPphrases ,7834 6 ,8089 27 ,6399 7 ,6397 ,7200 ,4850 ,7124 ,5312
mheilman/task6-ETS-TERp ,4477 73 ,7291 68 ,5253 57 ,5049 ,5217 ,4748 ,6169 ,4566
nitish_aggarwal/task6-aggarwal-run1** ,5777 52 ,8158 20 ,5466 52 ,3675 ,8427 ,3534 ,6030 ,4430
nitish_aggarwal/task6-aggarwal-run2** ,5833 51 ,8183 17 ,5683 42 ,3720 ,8330 ,4238 ,6513 ,4499
nitish_aggarwal/task6-aggarwal-run3 ,4911 67 ,7696 57 ,5377 53 ,5320 ,6874 ,4514 ,5827 ,2818
nmalandrakis/task6-DeepPurple-DeepPurple_hierarchical ,6228 43 ,8100 26 ,5979 23 ,5984 ,7717 ,4292 ,6480 ,3702
nmalandrakis/task6-DeepPurple-DeepPurple_sigmoid ,5540 56 ,7997 41 ,5558 50 ,5960 ,7616 ,2628 ,6016 ,3446
nmalandrakis/task6-DeepPurple-DeepPurple_single ,4918 66 ,7646 58 ,5061 61 ,4989 ,7092 ,4437 ,4879 ,2441
parthapakray/task6-JU_CSE_NLP-Semantic_Syntactic_Approach ,3880 82 ,6706 86 ,4111 76 ,3427 ,3549 ,4271 ,5298 ,4034
rada/task6-UNT-CombinedRegression ,7418 14 ,8406 7 ,6159 14 ,5032 ,8695 ,4797 ,6715 ,4033
rada/task6-UNT-IndividualDecTree ,7677 9 ,8389 9 ,5947 25 ,5693 ,8688 ,4203 ,6491 ,2256
rada/task6-UNT-IndividualRegression ,7846 5 ,8440 6 ,6162 13 ,5353 ,8750 ,4203 ,6715 ,4033
sbdlrhmn/task6-sbdlrhmn-Run1 ,6663 23 ,7842 53 ,5376 54 ,5440 ,7335 ,3830 ,5860 ,2445
sbdlrhmn/task6-sbdlrhmn-Run2 ,4169 78 ,7104 77 ,4986 64 ,4617 ,4489 ,4719 ,6353 ,4353
sgjimenezv/task6-SOFT-CARDINALITY ,7331 15 ,8526 5 ,6708 3 ,6405 ,8562 ,5152 ,7109 ,4833
sgjimenezv/task6-SOFT-CARDINALITY-ONE-FUNCTION ,7107 19 ,8397 8 ,6486 6 ,6316 ,8237 ,4320 ,7109 ,4833
siva/task6-DSS-alignheuristic ,5253 60 ,7962 42 ,6030 21 ,5735 ,7123 ,4781 ,6984 ,4177
siva/task6-DSS-average ,5490 58 ,8047 35 ,5943 26 ,5020 ,7645 ,4875 ,6677 ,4324
siva/task6-DSS-wordsim ,5130 61 ,7895 49 ,5287 55 ,3765 ,7761 ,4161 ,5728 ,3964
skamler_/task6-EHU-RUN1v2** ,3129 86 ,6935 83 ,3889 79 ,3605 ,5187 ,2259 ,4098 ,3465
sokolov/task6-LIMSI-cosprod ,6392 37 ,7344 67 ,3940 78 ,3948 ,6597 ,0143 ,4157 ,2889
sokolov/task6-LIMSI-gradtree ,6789 22 ,7377 66 ,4118 75 ,4848 ,6636 ,0934 ,3706 ,2455
sokolov/task6-LIMSI-sumdiff ,6196 45 ,7101 78 ,4131 74 ,4295 ,5724 ,2842 ,3989 ,2575
spirin2/task6-UIUC-MLNLP-Blend ,4592 70 ,7800 56 ,5782 35 ,6523 ,6691 ,3566 ,6117 ,4603
spirin2/task6-UIUC-MLNLP-CCM ,7269 16 ,8217 16 ,6104 17 ,5769 ,8203 ,4667 ,5835 ,4945
spirin2/task6-UIUC-MLNLP-Puzzle ,3216 85 ,7857 51 ,4376 69 ,5635 ,8056 ,0630 ,2774 ,2409
sranjans/task6-sranjans-1 ,6529 30 ,8018 39 ,6249 12 ,6124 ,7240 ,5581 ,6703 ,4533
sranjans/task6-sranjans-2 ,6651 24 ,8128 22 ,6366 8 ,6254 ,7538 ,5328 ,6649 ,5036
sranjans/task6-sranjans-3 ,5045 62 ,7846 52 ,5905 30 ,6167 ,7061 ,5666 ,5664 ,3968
tiantianzhu7/task6-tiantianzhu7-1 ,4533 72 ,7134 74 ,4192 73 ,4184 ,5630 ,2083 ,4822 ,2745
tiantianzhu7/task6-tiantianzhu7-2 ,4157 80 ,7099 79 ,3960 77 ,4260 ,5628 ,1546 ,4552 ,1923
tiantianzhu7/task6-tiantianzhu7-3 ,4446 74 ,7097 80 ,3740 81 ,3411 ,5946 ,1868 ,4029 ,1823
weiwei/task6-weiwei-run1** ,6946 20 ,8303 10 ,6081 19 ,4106 ,8351 ,5128 ,7273 ,4383
yeh/task6-SRIUBC-SYSTEM1 ,7513 11 ,8017 40 ,5997 22 ,6084 ,7458 ,4688 ,6315 ,3994
yeh/task6-SRIUBC-SYSTEM2 ,7562 10 ,8111 24 ,5858 33 ,6050 ,7939 ,4294 ,5871 ,3366
yeh/task6-SRIUBC-SYSTEM3 ,6876 21 ,7812 54 ,4668 68 ,4791 ,7901 ,2159 ,3843 ,2801
ygutierrez/task6-UMCC_DLSI-MultiLex ,6630 26 ,7922 46 ,5560 49 ,6022 ,7709 ,4435 ,4327 ,4264
ygutierrez/task6-UMCC_DLSI-MultiSem ,6529 29 ,8115 23 ,6116 16 ,5269 ,7756 ,4688 ,6539 ,5470
ygutierrez/task6-UMCC_DLSI-MultiSemLex ,7213 18 ,8239 14 ,6158 15 ,6205 ,8104 ,4325 ,6256 ,4340
yrkakde/task6-yrkakde-DiceWordnet ,5977 50 ,7902 48 ,5742 39 ,5294 ,7470 ,5531 ,5698 ,3659
yrkakde/task6-yrkakde-JaccNERPenalty ,6067 47 ,8078 31 ,5955 24 ,5757 ,7765 ,4989 ,6257 ,3468

 

Notes

* system submitted past the 120 hour window
** post-deadline fixes

Changes

Some teams had several issues which are now solved:
- davide_buscaldi: a bug in the treatment of NaNs
- janardhan: a bug in the treatment of NaNs
- nmalandrakis: results were missed because of wrong naming
- sgjimenezv: results had been shifted because of wrong naming
- nitish_aggarwal: results had been shifted because of wrong naming

The following teams submitted post-deadline fixes:
- weiwei
- nitish_aggarwal
- skamler_