Task Description


The Cross-Lingual Textual Entailment task (CLTE) addresses textual entailment (TE) recognition under a new dimension (cross-linguality), and within a new challenging application scenario (content synchronization).


Given a pair of topically related text fragments (T1 and T2) in different languages, the CLTE task consists of automatically annotating it with one of the following entailment judgments:


  •  Bidirectional (T1 ->T2 & T1 <- T2): the two fragments entail each other (semantic equivalence)
  • Forward (T1 -> T2 & T1 !<- T2): unidirectional entailment from T1 to T2
  • Backward (T1 !-> T2 & T1 <- T2): unidirectional entailment from T2 to T1
  • No Entailment (T1 !-> T2 & T1 !<- T2): there is no entailment between T1 and T2


In this task, both T1 and T2 are assumed to be TRUE statements; hence in the dataset there are no contradictory pairs.




<entailment-corpus  languages="spa-eng">
          <pair id=“1” entailment=“bidirectional”>
                    <t1>Mozart nació en la ciudad de Salzburgo</t1>
                    <t2>Mozart was born in Salzburg.</t2>
          <pair id=“2” entailment="forward”>
 <t1>Mozart nació el 27 de enero de 1756 en Salzburgo</t1>
                    <t2> Mozart was born in 1756 in the city of Salzburg.</t2>

          <pair id=“3” entailment="backward”>

                    <t1>Mozart nació en la ciudad de Salzburgo</t1>
                    <t2>Mozart was born on 27th January 1756 in Salzburg.</t2>                    

          <pair id=“4” entailment="no_entailment”>
                    <t1>Mozart nació el 27 de enero de 1756 en Salzburgo</t1>
                    <t2>Mozart was born to Leopold and Anna Maria Pertl Mozart.</t2>


The proposed task represents an application-oriented variant of the traditional Recognizing Textual Entailment (RTE) task [1,2], targeting the cross-lingual content synchronization scenario proposed in [3,4].


The dataset consists  of CLTE pairs, balanced with respect to the four entailment judgments (bidirectional, forward, backward, and no entailment). Cross-lingual datasets are available for the following language combinations, i.e. Spanish/English, German/English, Italian/English and French/English.


For further information please refer to the Task Guidelines.

If you are interested in the task, please join the CLTE discussion group.

You can register to the task at the SemEval 2013 website.



Why cross-lingual Textual Entailment?

 Cross-linguality represents a dimension of the TE recognition problem that so far has been only partially investigated. The great potential of integrating monolingual TE recognition components into NLP architectures has been reported in several areas, including question answering, information retrieval, information extraction, and document summarization. However, mainly due to the absence of CLTE recognition components, similar improvements have not been achieved yet in any cross-lingual application. The CLTE task aims at prompting research to fill this gap. Along such direction, research can now benefit from recent advances in other fields, especially machine translation (MT), and the availability of: i) large amounts of parallel and comparable corpora in many languages, ii) open source software to compute word-alignments from parallel corpora, and iii) open source software to set-up strong MT baseline systems. We believe that all these resources can potentially help in developing inference mechanisms on multilingual data.


Why cross-lingual content synchronization?

The explosion of multilingual user-generated content in websites like Wikipedia provides users with the opportunity to access information about a given topic in their own language. However, to take full advantage of this opportunity, it would be important to present the user with the same content, independently from the language of the article. Currently, to address this issue, multilingual Wikis rely on contributors to manually translate different pages on the same subject. This is not only a time-consuming procedure but also the source of many inconsistencies, as contributors update the different language versions separately, and every update would require translators to compare the different language versions and synchronize the updates. These problems, which cannot be faced by asking contributors to adhere to restrictive content creation guidelines, represent an interesting direction for research on automated solutions. Content synchronization represents a challenging application scenario to test the capabilities of advanced NLP systems. Given two documents about the same topic written in different languages (e.g. Wikipedia articles), the task consists of automatically detecting and resolving differences in the information they provide, in order to produce aligned, mutually enriched versions of the two documents. Towards this objective, a crucial requirement is to identify the information in one page that is equivalent or novel (more informative) with respect to the content of the other. The task can be naturally cast as an entailment-related problem, where bidirectional and unidirectional entailment judgments for two text fragments are respectively mapped into judgments about semantic equivalence and novelty. Alternatively, the task can be seen as a Machine Translation evaluation problem, where judgments about semantic equivalence and novelty depend on the possibility to fully or partially translate a text fragment into the other.


Potential interest


The two main differences with respect to the evaluation framework proposed by the RTE Challenges are: i) the multilingual dimension, and ii) the multi-directionality of the entailment relations. It is worth noting that, despite such differences, the CLTE task can be easily addressed by the community of RTE participants with a minimum effort. On the one hand, the multilinguality issue can be bypassed by translating the non-English element of each pair into English. On the other hand, the CLTE dataset can be easily transformed into a traditional RTE corpus by duplicating and swapping the T1 and T2 elements of each pair. In light of these considerations, we believe that all groups taking part to the RTE evaluation exercise (13 in the last edition) would be potentially interested also in the CLTE task.

Besides the recent advances on monolingual TE, also the methodologies used in Statistical Machine Translation (SMT) offer promising solutions to approach the CLTE task. In line with a number of systems that model the RTE task as a similarity problem (i.e. handling similarity scores between T and H as useful evidence to draw entailment decisions), the standard sentence and word alignment programs used in SMT offer a strong baseline for CLTE. In light of this consideration, also the MT community would be potentially interested in the proposed challenge. However, it has to be remarked that, although representing a solid starting point to approach the problem, similarity-based techniques are just approximations, open to significant improvements coming from semantic inference at a multilingual level (e.g. cross-lingual entailment rules such as “perro”→“animal”). Taken in isolation, similarity-based techniques clearly fall short from providing an effective solution to the problem of assigning directions to the entailment relations (especially in the complex CLTE scenario, where entailment relations are multi-directional). 


Altogether, the mentioned differences with respect to the traditional RTE Challenges, and the contiguity between CLTE and TE and SMT provide: i) strong motivations to participate in the challenge to a potentially large community of researchers coming from different areas, ii) solid starting points to approach the problem from different perspectives, and iii) large room for mutual improvement.


Task organizers 


  • Matteo Negri,  FBK-irst (Trento, Italy), negri [at] fbk.eu
  • Yashar Mehdad, University of British Columbia (Vancouver, B.C., Canada), ymahdad [at] gmail.com
  • Luisa Bentivogli, FBK-irst (Trento, Italy), bentivo [at] fbk.eu
  • Danilo Giampiccolo, CELCT (Italy), giampiccolo [at] celct.it
  • Alessandro Marchetti, CELCT (Italy), amarchetti [at] celct.it




  • November 1, 2012: Full Training Data available for participants
  • February 15, 2013: Registration Deadline [for Task Participants]
  • March 1, 2013: Test data release
  • March 8, 2013: Task submissions deadline
  • March 15, 2013: Release of individual results
  • April 9, 2013:  Paper submission deadline [TBC]
  • April 23, 2013: Reviews Due [TBC]
  • April 29, 2013: Camera ready Due 
  • June 14-15, 2013 [TBC]: SemEval Workshop associated with *Sem Conference (June 13-14, 2013 TBC) co-located with  NAACL HTL 2013, Atlanta, Georgia, USA




[1]  L. Bentivogli,  I. Dagan, H. T.  Dang, D. Giampiccolo, B. Magnini. 2009. The Fifth PASCAL Recognizing Textual Entailment Challenge. In TAC 2009 Workshop Proceedings, NIST, Gaithersburg, MD, USA.

[2] L. Bentivogli, P. Clark, I. Dagan, H. T. Dang, and D. Giampiccolo. 2010. The Sixth PASCAL Recognizing Textual Entailment Challenge. In TAC 2010 Workshop Proceedings, NIST, Gaithersburg, MD, USA.

[3] Y. Mehdad, M. Negri, and M. Federico. 2010. Towards Cross-Lingual Textual Entailment. In Proceedings of NAACL-HLT 2010.

[4] Y. Mehdad, M. Negri, and M. Federico. 2011. Using Parallel Corpora for Cross-lingual Textual Entailment. In Proceedings of ACL-HLT 2011.

[5] M. Negri, L. Bentivogli, Y. Mehdad, D. Giampiccolo, and A. Marchetti. 2011. Divide and Conquer: Crowdsourcing the Creation of Cross-Lingual Textual Entailment Corpora. In Proceedings of EMNLP 2011.

[6] M. Negri,  A. Marchetti, Y. Mehdad, L. Bentivogli and D. Giampiccolo, 2012. Semeval-2012 Task 8: Cross-lingual Textual Entailment for Content Synchronization. In Proceedings of *SEM 2012, .pdf


Contact Info


Other Info


  • Nov. 23, 2012: First Call for Participation
  • Sep. 1, 2012: Registration to SemEval 2013 open
  • Aug. 17, 2012: Trial data available