Task Description

One of the essential functions of natural language is to talk about spatial relationships between objects. Linguistic constructs can express highly complex, relational structures of objects, spatial relations between them, and patterns of motion through space relative to some reference point. Understanding such spatial utterances is a problem in many areas, including robotics, navigation, traffic management, and query answering systems. Learning how to map this information onto a formal representation from text is a challenging problem. At present no well-defined framework for automatic spatial information extraction exists that can handle all of these issues. The task of spatial role labeling (Kordjamshidi et al. 2010) introduces an annotation scheme proposed that is language-independent and facilitates the application of machine learning techniques. The framework consists of a set of spatial roles based on the theory of holistic spatial semantics with the intent of covering all aspects of spatial concepts, including both static and dynamic spatial relations. Let us illustrate the spatial role labeling task with the following example:

 


For the sentence: Give me the gray book on the big table.

Spatial role labeling results in the following output: Give me [the gray book]TRAJECTOR [on]SPATIAL_INDICATOR [the big table]LANDMARK.
 

The phrase headed by the token book is referring to a trajector object, the phrase headed by the token table is referring to the role of a landmark and these are related by the spatial expression on denoted as spatial indicator. The spatial indicator (often a preposition) establishes the type of spatial relation.

Analogous to semantic role labeling we define spatial role labeling as the task of automatic labeling of words and phrases in a sentence with a set of spatial roles such as trajector, landmark, spatial indicator, distance, direction etc. More specifically, the task involves identifying and classifying spatial arguments that are triggered by spatial expressions mentioned in a sentence and establishing relations between them with attributes of holistic spatial semantics.

For Semantic Evaluations 2013 the Spatial Role Labeling task has been largely derived from the SemEval-2012 task (Kordjamshidi et al. 2010, 2012). In contrast to SpRL at SemEval-2012, a broader set of spatial annotations has been introduced, including paths, directions and distances, as well as the concept of motion, as performed by the trajector along the path (Mani and Pustejovsky, 2012), has been proposed. In the following sections we describe the major concepts of spatial role labeling with examples and annotation formats, as well as the tasks, evaluation criteria and metrics.

 

Spatial Role Labels

Spatial role labels (annotations are highlighted by [] with corresponding textual spans and labels in subscript) are defined below along with their Backus-Naur forms (BNF).

 

Trajector

Trajector is a spatial role label assigned to a word or a phrase that denotes a central object of a spatial scene or an object that moves.

Examples:

1)      ... [the book] TRAJECTOR on the desk ...

2)      ... [he] TRAJECTOR passed the village ...

 

The annotation of trajectors is implemented by an XML tag <TRAJECTOR> and specified by the following BNF:
attributes ::= id start end [text] [comment]
id ::= ID
{id ::= Trajector_id
Trajector_id ::= T<integer>}
start ::= integer
end ::= integer
text ::= CDATA
comment ::= CDATA

where id is a unique identifier of the annotation; start is the character offset which indicates the start position of the annotation from the beginning of the document; end is the character offset which indicates the end position of the annotation from the end of the document; text is the textual content of the annotation; and, comment is the annotator's comment to the annotation.

 

Landmark

Landmark is a spatial role label assigned to a word or a phrase that denotes a secondary object of a spatial scene, to which a possible spatial relation (as between two objects in space) can be established.

Examples:

1)      ... the book on [the desk] LANDMARK ...

2)      ... he passed [the village] LANDMARK ...

3)      In [Brasil] LANDMARK I started from ...

 

The annotation of landmarks is implemented by an XML tag <LANDMARK> and specified by the following BNF:
attributes ::= id start end [text] [comment]
id ::= ID
{id ::= Landmark_id
Landmark_id ::= L<integer>}
start ::= integer
end ::= integer
text ::= CDATA
comment ::= CDATA

where id is a unique identifier of the annotation; start is the character offset which indicates the start position of the annotation from the beginning of the document; end is the character offset which indicates the end position of the annotation from the end of the document; text is the textual content of the annotation; and, comment is the annotator's comment to the annotation.

 

Spatial Indicator


Spatial indicator is a spatial role label assigned to a word or a phrase that signals a spatial relation between objects (trajectors and landmarks) of a spatial scene.

Examples:

1)      ... the book [on] SPATIAL_INDICATOR the desk ...

2)      [In] SPATIAL_INDICATOR Brasil I started from ...

The annotation of spatial indicators is implemented by an XML tag <SPATIAL_INDICATOR> and specified by the following BNF:
attributes ::= id start end [text] [comment]
id ::= ID
{id ::= Spatial_indicator_id
Spatial_indicator_id ::= S<integer>}
start ::= integer
end ::= integer
text ::= CDATA
comment ::= CDATA

where id is a unique identifier of the annotation; start is the character offset which indicates the start position of the annotation from the beginning of the document; end is the character offset which indicates the end position of the annotation from the end of the document; text is the textual content of the annotation; and, comment is the annotator's comment to the annotation.

 

Motion Indicator

Motion Indicator is a spatial role label assigned to a word or a phrase that signals a motion of the trajector of a spatial scene, and which is usually performed along a path.

Examples:

1)      He [went] MOTION_INDICATOR to...

2)      I [continued] MOTION _INDICATOR to ...

3)      After [leaving] MOTION _INDICATOR the motorway E45 ...

The annotation of motion indicators is implemented by an XML tag <MOTION_INDICATOR> and specified by the following BNF:
attributes ::= id start end [text] [comment]
id ::= ID
{id ::= Motionl_indicator_id
Motion_indicator_id ::= M<integer>}
start ::= integer
end ::= integer
text ::= CDATA
comment ::= CDATA

where id is a unique identifier of the annotation; start is the character offset which indicates the start position of the annotation from the beginning of the document; end is the character offset which indicates the end position of the annotation from the end of the document; text is the textual content of the annotation; and, comment is the annotator's comment to the annotation.

 

Path

Path is a spatial role label assigned to a word or a phrase that denotes a path along which the trajector motion is being or have been performed. Please note that Path is a complex concept which may denote a route of motion (e.g. ... along the road, ... on E40 etc), a starting point of motion (he left the office), an end point of motion (he arrived in New York) , containing both start and end points of motion (he left Boston for New York), or a landmark traversed in motion (he crossed the bridge). In such complex paths multiple trajectors and landmarks can be identified, and which can occur in other spatial relations than the path with corresponding labels as trajectors and landmarks (see Example 5 (a and b) in Section Spatial Relation).

Examples:

1)      After leaving [the motorway E45] PATH ...

2)      He arrived in [New York] PATH ...

3)      He left [Boston for New York] PATH ...

4)      ... coming [from the streets 443 and 445] PATH ...

The annotation of paths is implemented by an XML tag <PATH> and specified by the following BNF:
attributes ::= id start end [text] [comment]
id ::= ID
{id ::= Path_id
Path_id ::= P<integer>}
start ::= integer
end ::= integer
text ::= CDATA
comment ::= CDATA

where id is a unique identifier of the annotation; start is the character offset which indicates the start position of the annotation from the beginning of the document; end is the character offset which indicates the end position of the annotation from the end of the document; text is the textual content of the annotation; and, comment is the annotator's comment to the annotation.

 

Direction

Direction is a spatial role label assigned to a word or a phrase that denotes an absolute or relative direction of motion, or a spatial arrangement between a trajector and a landmark.

Examples:

1)      ... located two kilometres [west] DIRECTION of the village ...

2)      ... reached a farm in [the south] DIRECTION ...

The annotation of directions is implemented by an XML tag <DIRECTION> and specified by the following BNF:
attributes ::= id start end [text] [comment]
id ::= ID
{id ::= Direction_id
Direction_id ::= DIR<integer>}
start ::= integer
end ::= integer
text ::= CDATA
comment ::= CDATA

where id is a unique identifier of the annotation; start is the character offset which indicates the start position of the annotation from the beginning of the document; end is the character offset which indicates the end position of the annotation from the end of the document; text is the textual content of the annotation; and, comment is the annotator's comment to the annotation.

 

Distance

Distance is a spatial role label assigned to a word or a phrase that denotes an absolute or relative distance of motion, or the distance between a trajector and a landmark.

Examples:

1)      ... located [two kilometres] DISTANCE west of the village ...

2)      ... [nearby] DISTANCE a small bridge ...

The annotation of distances is implemented by an XML tag <DISTANCE> and specified by the following BNF:
attributes ::= id start end [text] [comment]
id ::= ID
{id ::= Distance_id
Distance_id ::= DIS<integer>}
start ::= integer
end ::= integer
text ::= CDATA
comment ::= CDATA

where id is a unique identifier of the annotation; start is the character offset which indicates the start position of the annotation from the beginning of the document; end is the character offset which indicates the end position of the annotation from the end of the document; text is the textual content of the annotation; and, comment is the annotator's comment to the annotation.

 

Spatial Relation

Spatial Relation is an annotation of a spatial relation that hold between spatial markables in text (e.g. between a trajector and a landmark) (static relations), or between a trajector and a path with an explicit or implicit mentions for motion, direction or distance (dynamic relations). Each spatial relation is further classified into one of the following semantic classes derived from the work of Stock (1997) as: REGIONAL, DIRECTIONAL and DISTAL.

Examples:

1)      Text: ... the farm in the south of the CP ...

Annotations: ... [the farm] TRAJECTOR [in] SPATIAL_INDICATOR [the south] DIRECTION of [the CP] LANDMARK ...

Relation semantic type: DIRECTIONAL

 

2)      Text: ... I parked the car at the paved road ...

Annotations: ... I parked [the car] TRAJECTOR [at] SPATIAL_INDICATOR [the paved road] LANDMARK ...

Relation semantic type: REGIONAL

 

3)      Text: After leaving the motorway E45 I had to ...

Annotations: After [leaving] SPATIAL_INDICATOR [the motorway E45] PATH [I ] TRAJECTOR had to ...

Relation semantic type: DIRECTIONAL

 

4)      Text: ... I passed the village Bøgvad, followed the way "Mosevej" to the north ...

Annotations: ... [I] TRAJECTOR [passed] MOTION_INDICATOR [the village Bøgvad, followed the way "Mosevej"] PATH to [the north] DIRECTION ...

Relation semantic type: DIRECTIONAL

5)      Text: We ... drove through the villages outside town into Mountain Ridge Park.

Annotations:

a) [We] TRAJECTOR ... [drove] MOTION_INDICATOR [through the villages outside town into Mountain Pine Ridge Park] PATH.

Relation semantic type: DIRECTIONAL

b) ... through [the villages] TRAJECTOR [outside] SPATIAL_INDICATOR [town] LANDMARK

Relation semantic type: REGIONAL

 

The annotation of spatial relations is implemented by an XML tag <RELATION> and specified by the following BNF:
attributes ::= id trajector_id [landmark_id] [spatial_indicator_id] [motion_indicator_id] [path_id] [direction_id] [distance_id] [general_type]
id ::= ID
{id ::= Relation_id
Relation_Id ::= SR<integer>}
trajector_id ::= IDREF
{trajector_id ::= Trajector_id}
landmark_id ::= IDREF
{landmark_id ::= Landmark_id}
spatial_indicator_id ::= IDREF
{spatial_indicator_id ::= Spatial_indicator_id}
motion_indicator_id ::= IDREF
{motion_indicator_id ::= Motion_indicator_id}
path_id ::= IDREF
{path_id ::= Path_id}
direction_id ::= IDREF
{direction_id ::= Direction_id}
distance_id ::= IDREF
{distance_id ::= Distance_id}
general_type ::= 'REGIONAL' | 'DIRECTIONAL' | 'DISTAL'
comment ::= CDATA

where id is a unique identifier of the annotation; trajector_id is the reference for a trajector annotation by its unique identifier; landmark_id is the reference for a landmark annotation by its unique identifier; spatial_indicator_id is the reference for a spatial indicator annotation by its unique identifier; motion_indicator_id is the reference for a motion indicator annotation by its unique identifier; path_id is the reference for a path annotation by its unique identifier; direction_id is the reference for a direction annotation by its unique identifier; distance_id is the reference for a distance annotation by its unique identifier; general_type is the (general) semantic type of the spatial relation with respect to (Stock, 1997); and, comment is the annotator's comment to the annotation.

 

Annotations and data formats

The data for the shared task comprises two different corpora. The first corpus is a subset of the IAPR TC-12 image benchmark corpus; the second corpus comes from the Confluence project that targets the description of locations situated at each of the latitude and longitude integer degree intersection in the world. In general the data for the task will be provided in the following format:


<?xml version="1.0" encoding="UTF-8" ?>
<SpRL>
<TEXT><![CDATA[
http://confluence.org/confluence.php?id=18559
 
15'N 23'W
 
01-May-2003 -- On our way from Recife to Königsberg/Kaliningrad the next confluence after the Archipelago de Fernando de Noronha is in the Cape Verde Islands Archipelago.
 
This time we decided to proceed first to 15N 23W, which is fairly close to Ilha do Maio.
 
The wind was quite strong (force Beaufort 6-7 from NNE) and the visibility was rather limited. So we went a little closer to make some better pictures.
 
The North part of Ilha do Maio is low and mainly occupied by salt pans which are surrounded by sand dunes. In the South thre are several peaks amongst which is Monte Penoso, 435 m (1,430 ft) high and the highest point of Ilha do Maio. The islands slopes gradually from these central peaks to the South end. The coasts of Ilha do Maio are, in general, low. The East coast is more indented than the other ones. The soil is for the most part dry and sterile. Salt, which is exported out of the archipelago, is the only produce of importance. There are also two limeworks and a fish canning factory. Landing on Ilha do Maio is bad and impeded with swell. There is no regular communication by sea. A small landing strip is suitable for light aircraft by day.
]]></TEXT>
<TAGS>
<TRAJECTOR id="T0" start="-1" end="-1" text="" />
<TRAJECTOR id="T1" start="273" end="280" text="15N 23W" />
<TRAJECTOR id="T2" start="420" end="422" text="we" />
<TRAJECTOR id="T3" start="475" end="489" text="The North part" />
<TRAJECTOR id="T4" start="475" end="505" text="The North part of Ilha do Maio" />
<TRAJECTOR id="T6" start="546" end="551" text="which" />
<TRAJECTOR id="T11" start="782" end="792" text="The coasts" />
<LANDMARK id="L0" start="307" end="319" text="Ilha do Maio" />
<LANDMARK id="L1" start="493" end="505" text="Ilha do Maio" />
<LANDMARK id="L2" start="536" end="545" text="salt pans" />
<LANDMARK id="L3" start="570" end="580" text="sand dunes" />
<LANDMARK id="L5" start="744" end="763" text="these central peaks" />
<LANDMARK id="L6" start="796" end="808" text="Ilha do Maio" />
<SPATIAL_INDICATOR id="S0" start="304" end="306" text="to" />
<SPATIAL_INDICATOR id="S1" start="490" end="492" text="of" />
<SPATIAL_INDICATOR id="S2" start="524" end="535" text="occupied by" />
<SPATIAL_INDICATOR id="S3" start="556" end="569" text="surrounded by" />
<SPATIAL_INDICATOR id="S4" start="618" end="625" text="amongst" />
<SPATIAL_INDICATOR id="S5" start="764" end="766" text="to" />
<SPATIAL_INDICATOR id="S6" start="793" end="795" text="of" />
<MOTION_INDICATOR id="M0" start="82" end="85" text="way" />
<MOTION_INDICATOR id="M1" start="423" end="427" text="went" />
<PATH id="P0" start="75" end="123" text="On our way from Recife to Königsberg/Kaliningrad" />
<PATH id="P2" start="428" end="443" text="a little closer" />
<DIRECTION id="DIR0" start="585" end="594" text="the South" />
<DIRECTION id="DIR1" start="767" end="780" text="the South end" />
<DISTANCE id="DIS0" start="298" end="303" text="close" />
<RELATION id="SR0" trajector_id="T0" path_id="P0" motion_id="M0" general_type="DIRECTION" />
<RELATION id="SR1" trajector_id="T1" landmark_id="L0" spatial_indicator_id="S0" distance_id="DIS0" general_type="DISTANCE"/>
<RELATION id="SR2" trajector_id="T2" path_id="P2" motion_id="M1" general_type="DIRECTION" />
<RELATION id="SR3" trajector_id="T3" landmark_id="L1" spatial_indicator_id="S1" general_type="REGION" />
<RELATION id="SR4" trajector_id="T3" landmark_id="L2" spatial_indicator_id="S2" general_type="REGION" />
<RELATION id="SR5" trajector_id="T6" landmark_id="L3" spatial_indicator_id="S3" general_type="REGION" />
<RELATION id="SR8" trajector_id="T11" landmark_id="L6" spatial_indicator_id="S6" general_type="REGION" />
</TAGS>
</SpRL>

 

 

Task

The Spatial Relation Labeling task is divided into the following sub-tasks:

  • Task A: Identification of markable spans for three types of spatial annotations such as trajector, landmark and spatial indicator.
  • Task B: Identification of tuples (triplets) that connect trajectors, landmarks and spatial indicators identified in Task A into spatial relations. That is, identification of spatial relations with three markables connected, without semantic relation classification.
  • Task C: Identification of markable spans for all spatial annotations such as trajector, landmark, spatial indicator, motion indicator, path, direction and distance.
  • Task D: Identification of n-tuples that connect spatial markables identified in Task C into spatial relations. That is, identification of spatial relations with as many participated markables as possible, without semantic relation classification.
  • Task E: Semantic classification of spatial relations identified in Task D.

 

Evaluation Criteria and Metrics

 

Systems will be evaluated for each of the tasks in terms of precision (P), recall (R) and F1-score:

  • P = {number of correct system annotations}/{number of system annotations}
  • R = {number of correct system annotations}/{number of gold annotations}
  • F1 = (2 * P * R)/(P + R)

The system output will be evaluated against the gold annotations, thus it must conform to the BNFs described above. For tasks A and C, the system annotations are markables: spans of text associated with markable types such as trajector or direction. A system annotation of a markable is considered correct if it matches both the span (character offsets) and the type of a gold annotation (only one system annotation is permitted to match each gold annotation). For tasks B and D, the system annotations are spatial relation tuples (of length 3 in task B, of length 3 to 5 in task D) of references to markable annotations. A system annotation of a spatial relation tuple is considered correct if it is the same length as the gold annotation and if each element (markable reference) in the system tuple matches each element (markable reference) in the gold tuple. A system markable reference is considered to match a gold markable reference if the referenced markables have the same spans (character offsets) and markable types. For task E, the system annotations are spatial relation tuples of length 3 to 5, along with "general_type" labels. A system annotation of a spatial relation is considered correct if the spatial relation tuple is correct under the evaluation of task D and the "general_type" of the system relation is the same as the "general_type" of the gold relation.

 

References

  1. P. Kordjamshidi, M.-F. Moens, and M. van Otterlo. 2010. Spatial Role Labeling: Task Definition and Annotation Scheme. In Calzolari, N. (Ed.), Khalid, C. (Ed.), Bente, M. (Ed.), Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10). Seventh conference on International Language Resources and Evaluation. European Language Resources Association (ELRA).
  2. Parisa Kordjamshidi, Martijn Van Otterlo, Marie-Francie Moens  (2011). Spatial role labeling: Towards extraction of spatial relations from natural language. ACM Transactions on Speech and Language Processing, 8 (3), article 4, 36 p.
  3. P. Kordjamshidi, S. Bethard and M.-F. Moens. 2012. SemEval-2012 Task 3: Spatial Role Labeling, Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval).
  4. I. Mani, and J. Pustejovsky, J. 2012. Interpreting Motion: Grounded Representations for Spatial Language. Explorations in language and space, Oxford University Press.
  5. O. Stock, editor. 1997. Spatial and Temporal Reasoning. Kluwer.

Contact Info

Organizers

  • Oleksandr Kolomiyets
    Katholieke Universiteit Leuven, Belgium [Primary Contact]
  • Parisa Kordjamshidi
    Katholieke Universiteit Leuven, Belgium
  • Steven Bethard
    University of Colorado, USA
  • Marie-Francine Moens
    Katholieke Universiteit Leuven, Belgium

Other Info

Announcements

  • Feb. 5, 2013: Training data available (Confluence Corpus)
  • Nov. 18, 2012: Training data available (CLEF Corpus)
  • Aug. 31, 2012: Task description is finalized