Published online first in:
Educational Researcher
May 15, 2014

Joseph P. Robinson-Cimpian, University of Illinois at Urbana-Champaign

Abstract

This article introduces novel sensitivity-analysis procedures for investigating and reducing the bias that mischievous responders (i.e., youths who provide extreme, and potentially untruthful, responses to multiple questions) often introduce in adolescent disparity estimates based on data from self-administered questionnaires (SAQs). Mischievous responders affect a wide range of disparity estimates, including those between adoptees and nonadoptees, sexual minorities and nonminorities, and individuals with and without disabilities. Thus, the procedures introduced here have broad relevance to research and can be widely, and easily, implemented. The sensitivity-analysis procedures are illustrated with SAQ data from youths in Grades 9–12 (N = 11,829) to examine between-group disparities based on sexual identity, gender identity, and physical disability. Sensitivity analyses revealed that each disparity estimated with these data was extremely sensitive to the presence of potentially mischievous responders. Patterns were similar across multiple approaches to dealing with mischievous responders, across various outcomes, and across different between-group comparisons. Mischievous responders are ubiquitous in adolescent research using SAQs and can, even in small numbers, lead to inaccurate conclusions that substantively affect research, policy, and public discourse regarding a variety of disparities. This article calls attention to this widespread problem and provides practical suggestions for assessing it, even when data are already collected.

• Read the full article
• Read the news release
  "Study: Addressing 'Mischievous Responders' Would Increase Validity of Adolescent Research"

Teens Are Shameless Liars When You Ask Them Serious Questions for Science
The Wire, May 23, 2014

'Mischievous Responders' Confound Research On Teens
NPR, May 22, 2014

'Mischievous Responders' May Skew Survey-Based Studies of Teens (subscription required)
Education Week, May 15, 2014