Problematic questions and missing dimensions

Engaging with Questions of Higher Order: cognitive vigilance required for higher degrees of twistedness (Part #10)

The vision and roadmap statements raise the question of whether they have framed the approach to questions in a manner that omits certain dimensions.

Problematic questions: Art Graesser's previous research had identified 12 problems with questions that frequently occur in surveys (Graesser, Bommareddy, Swamer, and Golding, 1996; Graesser, Kennedy, Wiemer-Hastings, and Ottati, 1999). Many of these problems have been incorporated in various analytical coding schemes of survey methodologists. Art Graesser, Katja Wiemer-Hastings and Roger Kreuz (The Gold Standard of Question Quality on Surveys) report on the development of a computer tool (QUAID: Question Understanding Aid) that assists survey methodologists who want to improve the wording, syntax, and semantics of questions on surveys.

At present, QUAID can handle 6 of these problems with some degree of correspondence with human experts, so we will focus on these 6 problems in the present paper. These problems are:

Unfamiliar technical term: There is a word or expression that very few respondents would know the meaning of.

Vague or imprecise predicate or relative term: The values of a predicate (i.e., main verb, adjective, or adverb) are not specified on an underlying continuum (e.g., try, large, frequently).

Vague or ambiguous noun-phrase: The referent of a noun-phrase, noun, or pronoun is unclear or ambiguous (e.g., items, amount, it, there).

Complex syntax: The grammatical composition is embedded, dense, structurally ambiguous, or not well-formed formed syntactically.

Working memory overload: Words, phrases, or clauses impose a high load on immediate memory.

Misleading or incorrect presupposition: The truth value of a presupposed proposition is false or /inapplicable.

Adam Stone (Typology of Questions Faced by Policymakers, 1996) notes that issues in which criminal courts perceive a role for regulation, involing "new" types of questions, pose problems which exacerbate many of the difficulties inherent in the policymaking role for courts generally -- and, by extension, for wider policymaking arenas. For example:

No Control Over Questions: Since courts cannot exercise control over the controversies presented to them, the cases by which they make policy may not be representative of the issues.
No Outside Sources: Courts generally lack resources and authority to conduct outside research into the often highly technical questions posed by new technologies.
Adversarial Science: The adversarial nature of court proceedings gives little weight to the best available science, instead pitting opposite extremes of scientific findings against each other as well as including the potential for biased experts and misleading questioning and opinions.
Lay People: Judges, juries, and lawyers are generally unqualified to devaluate highly technical and scientific claims. Further, since Daubert in 1992, the courts aind themselves evaluating more and more scientific evidence as the peer review standard has been further eroded.
Always Ex-post-facto: Courts always face questions in which alleged wrongs have already been committed, possibly on a wide scale. While legislatures sometimes, if not always, face the same situation, it is not inherent in their structure that they do so.
Decision by Analogy and Precedent: While this was discussed above and will be pursued in depth in the next section, it is important to keep in mind this crucial problem. Courts are, by all accounts, at least somewhat bound by precedent. Models differ in the extent to which this is perceived as a determining factor. Nevertheless, decisions on "new" issues under the shadow of precedents may prevent courts from developing to meet the demands of technological change. This is a non-normative statement though, as lack of adaptation by courts themselves may be perceived to be a good thing, allowing development of the law to be completed by democratically controlled legislatures.

These factors combine to create a court system which faces many ongoing and inherent problems when it must grapple with highly technical policymaking questions. Of course such potential confusion offers many opportunities for deliberately misleading strategies by those responsible for framing the questions.

Incorporating the temporal dimension: James Pustejovsky, Janyce Wiebe and Mark Maybury. (Multi-perspective and Temporal Question Answering, 2000) respond to the above question answering vision (Carbonell et al., 2000) and roadmap (Burger et al., 2002) with the concern:

Although a range of question and answer types are described, the ability to interpret a question and provide an answer with respect to different perspectives and the ability to answer questions involving temporal dimensions are largely unaddressed. This position paper argues for the importance of multiple perspective and temporal question answering and attempts to outline some aspects of the problem that would be important to capture on the Q-and-A roadmap.

Questions leading to deeper understanding: Benjamin Bloom (Taxonomy of educational objectives: the classification of educational goals, 1956) created a taxonomy for categorizing level of abstraction of questions that commonly occur in educational settings. Bloom's taxonomy of questions forces a person to interact with the text on a deeper level, and develop a more thorough understanding. Generally speaking, the complexity of questions can then be evaluated according to the type of mental process they entail. Their complexity also depends on the features of the question (any kind of ambiguity, degree of objectivity/subjectivity, implicit/explicit data, ...) and the features of the answer (nb of elements required, type of analysis required, ...) [more]. Bloom's taxonomy of questions offers an indicator of progress from simple questions of fact, through synthesis of multiple facts, through analysis based on synthesis, to a level of judgement or critical thinking:

Knowledge (simple recall): utilizing WH-questions (see below) to inform the client on the topic and then asking those questions to solidify knowledge.
Comprehension (familiar with meaning/can make use): utilizing concepts such as "explain, describe, compare, contrast, define, locate, match" to enable the client to explain and further understand the information.
Application (applying abstraction to a new example): now the client uses the information to apply it to new situations by responding to requests such as "show, what else, choose, sort, classify, name some other..., instead of..."
Analysis (break idea into constituent parts): now the client breaks down the information into parts and looks at how they relate back to the whole via concepts such as why, how, alike/different, order, identify, etc.
Synthesis (creation from existing elements or principles): at this step the client takes the knowledge and creates something new. Words of direction for this step might include: create, suppose, what if, write a story, draw a picture, make, think of, predict, and how many ways...
Evaluation (formation and substantiation of a judgment): the client assesses the information learned, states an opinion and critiques based on his/her previous knowledge. Words like: decide, judge, discuss, choose, recommend, give your opinion, state your preference, explain, will help them to complete this step.

Richard Paul (Critical Thinking: how to prepare students for a rapidly changing world, 1993) created a taxonomy of Socratic questions in support for problem based learning (PBL). The taxonomy is not a hierarchy in the traditional sense. The categories build upon each other, but they do not necessarily follow a pattern or design. One question's response will lead into another category of questioning not predetermined by the facilitator. In keeping with the problem based learning (PBL) philosophy, this aspect of the model is most conducive! The role of the skilled facilitator is to keep the inquiry "train on track", but, also, to allow participants to "travel to a viable destination" of their own design. Paul suggests six types of questions that probe reasons and evidence:

Questions of clarification
Questions that probe assumptions
Questions that probe reasons and evidence
Questions about viewpoints or perspectives
Questions that probe implications and consequences
Questions about the question

The particular function of critical thinking has been expressed as follows (from Questioning Deeply: Raising and Pursuing Root or Significant Questions):

Critical thinkers can pursue an issue in depth, covering various aspects in an extended process of thought or discussion. When reading a passage, they look for issues and concepts underlying the claims expressed. They come to their own understanding of the details they learn, placing them in the larger framework of the subject and their overall perspectives. They contemplate the significant issues and questions underlying subjects or problems studied. They can move between basic underlying ideas and specific details. When pursuing a line of thought, they are not continually dragged off the subject. They use important issues to organize their thought and are not bound by the organization given by another

Main menu

Directions

You are here

Main menu

Search form

Directions

You are here

Problematic questions and missing dimensions