[open-linguistics] First Call for Participation: SemEval - Task 4: TimeLine: Cross-Document Event Ordering (pilot task)

[Erp, M.G.J. van]

SemEval-2015 Task 4: TimeLine: Cross-Document Event Ordering (pilot task)
First Call for Participation 


Website: http://alt.qcri.org/semeval2015/task4 
Google Group: https://groups.google.com/forum/#!forum/semeval-task4-timeline
Evaluation period: November 15 - 30, 2014
Paper submission: January 2015 

*Introduction*
In any domain, professionals need to have access to knowledge in order to take well-informed decisions. An insightful way of presenting information in an easily updatable and complete manner is to present it on a timeline that is continuously updated with new information. The aim of the task is to build timelines from written news in English. More specifically, the goal is to order on a timeline all the events in which a target entity is involved. We focus mainly on cross-document event coreference resolution and cross-document temporal relation extraction.

Temporal relation extraction has been the topic of the three past TempEval tasks as part of SemEval:
	• TempEval-1 (2007): Temporal Relation Identification
	• TempEval-2 (2010): Evaluating Events, Time Expressions, and Temporal Relations
	• TempEval-3 (2013): Temporal Annotation
In addition, temporal relation extraction has been the focus of the 6th i2b2 NLP Challenge for clinical records but the cross-document aspect, has not been often explored. At RANLP 2009 there was a cross-document temporal relation extraction task, in which the goal was to link pre-defined events involving the same centroid entities (i.e. entities frequently participating in events) on a timeline. Nominal coreference resolution has been the topic of SemEval 2010 Task on Coreference Resolution in Multiple Languages.

Partially motivated by the work in the NewsReader project (http://www.newsreader-project.eu), TimeLine goes beyond the these tasks by addressing coreference resolution for events and temporal relation identification across documents.

*Task Description*
Given a set of documents and a target entity, the task is to build an event TimeLine related  to that entity, i.e. to detect, anchor in time and order the events involving the target entity.

As input data, we provide a set of documents and a set of target entities (people, organization, product or financial entity); only entities of interest will be selected as target entities, i.e. entities involved in many events across different documents and for which it is relevant to build a timeline. 

There are two tracks in this task based on the data used as input. For Track A only raw text is provided to the participants, while for Track B gold-standard event mentions are also given.

For both tracks the expected output is one TimeLine for each target entity. Each TimeLine consists of an ordered list of events in which each event is associated to a time anchor.

For both tracks a sub-track in which the events are not associated to a time anchor is proposed.

Participants can choose to participate in any track and subtrack.
Participants can submit up to two runs for each track/subtrack.

*Data*
The trial data consists of a set of 30 documents collected from Wikinews (http://en.wikinews.org) about Apple Inc. A set of target entities (input) and the corresponding ordered list of events (the output timeline) is provided with the set of documents.

The trial data have been annotated with the extents of event mentions and are available from http://alt.qcri.org/semeval2015/task4/index.php?id=data-and-tools. The evaluation tool can also be found there.

The evaluation data will consist of 3 sets of documents annotated with event mentions and a set of target entities. Each set will contain around 30 documents from Wikinews,  totalling around 30,000 tokens. For each set of documents, one file is provided containing the list of target entities.

No training corpus will be provided for this task.

*Evaluation Methodology*
Participants will submit the TimeLines produced by their system for all target entities. Systems will be ranked based on their temporal awareness.

*More information*
See http://alt.qcri.org/semeval2015/task4/ for further information about the task, data formats, examples, data downloads, tools and registration information.

*Important dates*
	• Evaluation period starts: November 15, 2014
	• Evaluation period ends: November 30, 2014
	• Paper submission due: January 30, 2015
	• Paper notification: Early March, 2015
	• Camera-ready due: March 30, 2015
	• SemEval workshop: Summer 2015

*Organisers*
	• Anne-Lyse Minard, Fondazione Bruno Kessler, Italy
	• Eneko Agirre, The University of the Basque Country, Spain
	• Itziar Aldabe, The University of the Basque Country, Spain
	• Marieke van Erp, VU University Amsterdam, Netherlands
	• Bernardo Magnini, Fondazione Bruno Kessler, Italy
	• German Rigau, The University of the Basque Country, Spain
	• Manuela Speranza, Fondazione Bruno Kessler, Italy
	• Rubén Urizar, The University of the Basque Country, Spain


--
Computational Lexicology & Terminology Lab (CLTL)
The Network Institute, VU University Amsterdam

De Boelelaan 1105
1081 HV  Amsterdam, The Netherlands
http://www.mariekevanerp.com
http://www.newsreader-project.eu