Researchers Urge Caution in Using Measures of Students’
"Non-Cognitive" Skills for Teacher Evaluation, School Accountability,
or Student Diagnosis 

WASHINGTON, D.C., May 13, 2015—Policymakers and practitioners have grown increasingly interested in measures of personal qualities other than cognitive ability—including self-control, grit, growth mindset, gratitude, purpose, emotional intelligence, and other beneficial personal qualities—that lead to student success. However, they need to move cautiously before using existing measures to evaluate educators, programs, and policies, or diagnosing children as having “non-cognitive” deficits, according to a review by Angela L. Duckworth and David Scott Yeager published in Educational Researcher, a peer-reviewed journal of the American Educational Research Association.

• VIDEO: Duckworth and Yeager discuss implications

Personal skills are generally considered to be characteristics that facilitate goal-directed effort, healthy social relationships, and sound judgment and decision making. Basic scientific research reliably shows that measures of these characteristics predict success in school, work, and life. Just a few survey questions, or one session of observing whether a child can delay gratification, can predict educational attainment, income, crime, and happiness months or years later. But these measures are generally not ready for educational use.

“Given the intense visibility and enthusiasm around growth mindset, grit, and other personal skills, it is important for school leaders and policymakers to realize that while there is great benefit to studying and assessing these attributes, the measures should not, currently, be used for broader accountability purposes,” said Duckworth, an associate professor of psychology at the University of Pennsylvania.

“Adapting and using these measures to decide whether a program is working for a school; how to promote or hire or fire teachers, principals, or staff; or how an institution can continually improve its practice and outcomes, are all different from what the measures were developed for,” said Yeager, an assistant professor of psychology at the University of Texas at Austin. “Measures developed for very good research purposes do not necessarily translate into these very important educational purposes.”

Duckworth and Yeager noted that:

• Measurement of personal qualities for school accountability purposes is a major topic in current conversations on Capitol Hill related to reauthorization of the Elementary and Secondary Education Act. “There are policy questions about whether states should be required or even encouraged to compare schools on their levels of these qualities,” said Yeager. “Our review says there is little or no scientific evidence that this should be done, and much evidence that it would yield misleading results.”
• Major federal education funders, like the Institute of Education Sciences and the Investing in Innovation Fund at the Department of Education, have included non-cognitive programs in their calls for proposals, requiring that program evaluation be carried out in many cases by examining changes in personal measures. “Our review provides direct guidance on how common measures will be misleading, and what kinds of measures will be better,” said Yeager. “This has direct application for the validity of the scientific database regarding personal characteristics and for the quality of education science.”
• School administrators are increasingly interested in using growth mindset measures for hiring or promoting teachers. “Such measures cannot yet be used reliably for the assessment of educators, nor are they suitable for between-school accountability judgments,” said Duckworth.

In their review, Duckworth and Yeager discussed the limitations and advantages of three common approaches to measuring personal qualities: self-report questionnaires administered to students, questionnaires administered to teachers about their students, and observation of student behavior on performance tasks.

“There really is no perfect measure for any aspect of personal skills. What we have are measures that have their distinct advantages and limitations. Developing better measures, and understanding which currently available measures are appropriate for which uses, are top priorities we should have as an education community,” said Duckworth.

“We advise practitioners and policymakers to seek out the most valid measure for their intended purpose(s),” wrote the authors. “Whenever possible, we recommend using a plurality of measurement approaches. While time and money are never as ample as would be ideal, a multi-method approach to measurement can dramatically increase reliability and validity.”

The authors encouraged further innovation in measurement development, including several “promising approaches” such as creating a suite of web-based, free, scalable “marshmallow test” tasks, or mining students’ online learning behavior or written communication in real time for meaningful patterns of behavior. With the right investments in research and development, Duckworth and Yeager suggested that personal measures could become valuable tools for purposes of program evaluation and educators’ improvement of classroom practices.

Funding Note
The review was made possible by grants to Duckworth from the National Institute on Aging, the Character Lab, the Bill and Melinda Gates Foundation, the Robert Wood Johnson Foundation, the Spencer Foundation, and the John Templeton Foundation, as well as grants to Yeager from the Raikes Foundation, the William T. Grant Foundation, and a fellowship from the Center for Advanced Study in the Behavioral Sciences.

About AERA
The American Educational Research Association (AERA) is the largest national professional organization devoted to the scientific study of education. Founded in 1916, AERA advances knowledge about education, encourages scholarly inquiry related to education, and promotes the use of research to improve education and serve the public good. Find AERA on Facebook and Twitter.

###