By Brittany Aguilar
A new study published today in PLoS ONE from researchers at the University of North Carolina-Chapel Hill demonstrates that some of the quantitative metrics used by graduate-admissions committees are poor predictors of productivity. Specifically, the authors suggest undergraduate grade-point average and Graduate Record Exam scores do not correlate with some standard metrics of Ph.D. graduate student success. Despite the importance of the application process, the authors state, “admissions committees should avoid over-reliance on any single component of the application and de-emphasize metrics that are minimally predictive of student productivity.”
The study analyzed graduate student productivity by examining the absolute number of publications, broken down by first and middle authorship, and time to degree. Among the students publishing, they were further binned into groups based on the number of papers published. The authors found neither time to degree nor authorship productivity were significantly correlated with GRE scores, scores on any subject GRE test, undergraduate GPA or previous research experience. To close followers of graduate education research, this may not be a surprising result: the predictive value of GRE scores have been in question since 1997. This led the National Institutes of Health to eliminate reporting of this metric on individual fellowship applications in 2015.
The only metric measured to show a strong correlation with authorship productivity was the mean recommender score–a qualitative measure of “Exceptional” through “Below Average”, as assigned by their recommenders–which could point towards a more holistic method of assessment. The authors speculate that “a [recommendation] letter writer will produce a generally positive letter for an above-average applicant, but will only select ’exceptional’ for those candidates who have shown the constellation of characteristics that typically correlate with research success.” Through this form of assessment, they claim, admissions committees may reveal students exhibiting traits critical to graduate success such as those who “persevere and maintain focus and optimism in the face of regular challenges”.
Despite the strong evidence pointing towards a lack of outcome prediction for GRE scores, the fact remains that admissions committees are under an enormous amount of pressure to quickly review and select students and, as such, have continued the practice of using GRE score cutoffs. Although the authors make no recommendations aside from a general movement towards “holistic assessment,” they include an interesting example of a non-traditional admissions practice in dental students. In the study, students were given an emotional intelligence exam prior to admission, testing for competencies in self-awareness, self-management, social awareness and relationship management. High scores on the EI test correlated significantly with high clinical performance and test scores, suggesting that there could be a relationship between emotional intelligence and dental student outcomes. Although the nature of clinical diagnostics differs from that of a laboratory setting, this example demonstrates there are alternative admissions methods that may provide insight to a person’s potential success in graduate school.
Completing a Ph.D. is a complex process that involves the contributions of many people. Thus, using authorship productivity to quantify graduate student success may oversimplify a combination of factors that graduate students must face. However, these metrics are valuable starting points as graduate students are often evaluated based on the number of publications and time-to-degree is a common metric used to assess departments.
This is part two of a two-part blog examining two papers published in PLoS ONE today. The other post is here.
—
Brittany Aguilar is interested in promoting public awareness of scientific advancements and encouraging stewardship of science within academia. She is currently completing her Ph.D. in Neuroscience at Georgetown University and is the founder of a graduate student organization, the Neuroscience Student Society, which serves to encourage dialogue in the areas of mentorship and graduate education. Brittany can be reached on Twitter (@brittaggie) or by email (ba438@georgetown.edu)
The RBR Writing Program is intended to help graduate students and postdocs receive policy writing experience. For more information, contact RBR Director Chris Pickett.