The objective of the second STEP meeting is to discuss future shared tasks in the area of computational semantics. The first STEP meeting (Venice, 2008) included a shared task on comparing semantic representation of texts as produced by natural language understanding systems. In the second STEP meeting at IWCS 2011, possible future shared task are presented and discussed. This STEP meeting, organised by Johan Bos, is the last event of IWCS-2011. There are three presentations followed by a panel discussion. It is free to attend for all IWCS participants, separate registration is not required.
The Deep Meaning Annotation Project (D-MAP) is concerned with the development of a large-scale deep semantic annotation of text and its application of machine learning methods in computational semantics. The focus is on integration of different aspects of meaning (word senses, thematic roles, quantifier scope, tense and aspect, anaphora, presupposition, rhetorical relations, background knowledge) into one semantic formalism. Such a resource doesn't exist yet, and we expect it to have a large impact on computational semantics because it will (a) enable quantified evaluation of semantic formalisms, (b) stimulate the use of statistical methods, and (c) create resources for applications such as NLG and MT.
Roser Morante: Annotating the scope of negation cuesIn this talk, I summarize the current achievements in processing the scope of negation cues, I describe briefly available resources, and focus on discussing issues related to the annotation of negation cues and their scope in The Hound of the Baskervilles, by Conan Doyle. Special attention will be given to discourse-level phenomena that influence the polarity of an statement in this type of texts, and that are not present in biomedical texts.
Anselmo Peñas: Question Answering for Machine Reading EvaluationWe present the "Question Answering for Machine Reading Evaluation" (QA4MRE) as a new task of the Question Answering (QA) Track at CLEF. The goal of this task is to evaluate Machine Reading abilities through Question Answering and Reading Comprehension Tests. This is related to other evaluation campaigns such as Recognizing Textual Entailment. We learned in these campaigns that the ability to make inferences about texts is correlated to the amount of knowledge considered. But this variable was never taken into account during evaluation, making very difficult to compare system technologies. We need to control the amount of knowledge required, but we don't want to build systems tuned for specific domains, but general technologies, able to self-adapt to new contexts or topics. The talk will discuss the decisions we took in QA4MRE to achieve this goal and to measure progress in two reading abilities at the same time: Capture knowledge from text collections and answer questions about a single text.