Question Generation Shared Task and Evaluation Campaign 2010 - Task B - QG from single sentences

A corpus of over 1000 questions (both human and machine generated). The automatically generated questions have been rated by several raters according to five criteria (relevance, question type, syntactic correctness and fluency, ambiguity, and variety).

The 1st Question Generation Shared Task & Evaluation Challenge took place in 2010 as part of the Third Workshop on Question Generation (http://www.questiongeneration.org/QG2010/). The aim of the challenge was to advance research on Question Generation (QG). The challenge consisted of two tasks: (A) Generation of Questions from Paragraphs, and (B) Generation of Questions from Single Sentences.

This dataset consists of the data from Task B: Generation from Sentences.

Systems that took part in this task were required to generate questions from single sentences. The participating teams were provided with an initial development data set (DevelopmentData_QuestionsFromSentences.xml) indicative of test data (TestData_QuestionsFromSentences.xml).

Both development and test data are in XML format. For the test data, the XML schema consisted of a root 'dataset' element with a child element called 'instance' for each test instance. The 'text' element contained the input to the QG system and the 'targetQuestionType' element specified the question type to be generated. Participant's submissions (anonymised as a, b, c, and d) are in the 'submission' element with the corresponding ratings for the quality of the generated questions (with multiple raters). The criteria associated with the ratings are described in InstructionsParticipants_QGfromSentences.pdf and in the associated conference proceedings paper available at http://oro.open.ac.uk/22345/..

Also included in this folder is the XSLT file used to translate the ratings into a table (evaluation.xsl). The table was exported to MS Excel to calculate various results. There is a spreadsheet (TestDataSummary.xls) which provides a view on the test data and results.