Testing & Evaluating new search interfaces

Hutchinson, H.B., Druin, A., Bederson, B. (2007) Supporting elementary-age children’s searching and browsing: Design and evaluation using the International Children’s Digital Library. JASIST, 58(11), 1618-1630.

General comments:

This article discusses the methodology and results of a series of user studies to assess the performance and preference of two different styles of interfaces (existing versus redesign). Although the results are not relevant to our discussion, this is a good example of a methodology that is frequently used to test a new interface or to prove its superiority over a previous design. The formula for these kinds of studies follows the same pattern: user search and browse tasks (both structured and unstructured), where quantitative measurements can be recorded (e.g. time/completion rates etc) followed by discussion/interview to gain more qualitative data about preference, ease of use, general thoughts etc.

Study methodology:

The International Children’s Library (ICDL) redesigned both the structure and presentation of its interface moving from a largely hierarchical structure to a more flat, faceted one. In order to evaluate the success of the changes, studies were designed to directly compare the new design to the old one. Children (the primary user group of the ICDL) in three different age groups were assigned to two different studies. In the simple study children were asked to conduct searches on both interfaces involving just a single category. In the second Boolean study, children were asked to conduct searches with each interface using two categories.

Tasks were divided in to four areas:

Browsing task:  A free-browsing task to observe how children used the interface without instruction.

Searching task: A series of six timed search tasks, such as “how many [imaginary creature] books are there?” (simple study) or “how many [imaginary creature and kid character] books are there? (Boolean study).

Comprehension task: In the Boolean study, children were asked to select two category buttons and then asked about the kinds of books they found. This was to find out whether children understood that selecting more than one button was creating a conjunctive Boolean search.

Subjective task/Interview: Upon completion of the other tasks, the children were asked if the interface was hard, medium or easy to use, and whether they liked using it not much, a little or a lot.

This entry was posted in Displays, Evaluation and tagged . Bookmark the permalink.