A Study of Some “Hard to Formulate” Biology
Working Note 33
Peter Clark, Boeing Research, June 2009
The below set of 22 questions was selected from the 50 final evaluation questions to
focus question formulation research during the 2009 Halo work on question-asking.
Specifically, we will be designing how domain knowledge, better dialog, and
paraphrasing can improve the question formulation process. The goal of this document
is not to design a solution, but to perform an in-depth analysis of some of the problems
that the solution will need to address.
Although the question-answering scores were relatively high in biology in the final
evaluation (typically in the 50%-70% range), and non-experts generally scored as high
as experts, the fidelity of the formulations was very low: on average, only 18% of the
words in the original question appeared in the CPL formulation, and, based on a
random sample, the users failed to formulate some aspect of the majority of questions
in the CPL version. There is thus substantial room for improvement in the formulation
of these questions.
The below set of 22 questions was selected by identifying those where the non-experts'
formulation received (on average) a lower1
score than the expert's formulation (using
the EE Biology KB). The goal was to select questions where the KB in principle could
answer the question (i.e., high expert score), but non-experts had difficulty (i.e., lower
non-expert score). In fact, the goal that the KB can in principle answer is only partially
met by this method - it turns out that in some cases the KB is unable to answer (or even
represent) the original question, yet the user obtained a good score by asking either a
slightly different or more general question. We give examples of this and how the users
were sometimes able to re-express questions in this way, and illustrate other problems
and solutions that arose. We finish with some concluding remarks about the phenomena
observed both in the original questions and the users’ formulation attempts.
Below, “EE expert” refers the (SRI) biology expert who posed questions against the
expert-built (Q1) KB. “EN non-experts” refer to the three non-experts (called Bio1,
Bio2, Bio3) who posed questions against the same KB. In particular we looked at the
questions posed by Bio2, who had the highest overall score of the three.
The Question Set
Which of the following is NOT a characteristic of a prokaryotic cell?