Developing a More Predictive Model for Identifying STEM Doctoral Students Who Are Likely to Succeed

There is now compelling data that GRE scores do not predict STEM doctoral completion and that admissions procedures that use these scores negatively impact diversity initiatives (see Appendix 1).  Our recent “If Not GRE Then What?” NEAGEP workshop at UMass Amherst on September 25-26th included over 60 faculty and administrators from 25 institutions and organizations.  Participants spent 1.5 days discussing alternatives to the GRE scores and results of the discussions are summarized below:

Session 1:  Identifying characteristics of successful STEM doctoral students

A) The characteristics listed below were those identified by the workshop participants as key (numbers in parentheses denote the number of times characteristics were identified by groups).

  1. Ability to ask questions, critical thinking, ability to listen and think about what they have heard, ability to reframe issues and look at things from multiple points of view, clarity of thought, capacity to reason, problem-solving ability, abstract thinking, ability to dig into an area, natural curiosity, inquisitive, common sense, can read the literature and find information, active thinker, ability to trouble-shoot using a logical approach (23)
  2. Commitment to something (job, hobby, activity), evidence of finishing things, ability to handle delayed gratification, positive persistence, tenacity, self-motivation, hardworking, resourceful (15)
  3. Adaptability, can learn from failure, experience with overcoming failure/adversity, overcoming obstacles, grit, resilience (11)
  4. Demonstrable relevant experience, discipline-specific experience, undergraduate research, research as a technician, work experience, past experience (8)
  5. Written and verbal communication skills (7)
  6. Responsible, self-assessment, self-awareness, self-inventory, ability to admit when you are wrong, truthful, honest (7)
  7. Independence, self-initiative, self-advocacy, autonomy (6)
  8. Evidence of near-peer networks, social and emotional intelligence, support system, ability to leverage social capitol (6)
  9. Creative thinking, creativity (5)
  10. Academic preparedness, knowledge and capability, quantitative skills, discipline-specific requirements (5)
  11. Clear reasons for wanting to get a PhD, knowing one’s interests, fit with the program to which they are applying, reasons for choosing a given institution, long-term career goals (5).
  12. Ability to work as a team, collaborative, team skills (5)
  13. Match for mentor, “mentorable”, teachable, confidence in mentor (4)
  14. Organizational and time management skills (3)

 

B) The characteristics listed below were identified by UMass NEAGEP ABD graduate students and alumni as important for their success. A list of 45 characteristics were rated by respondents and are rank-ordered based on perceived importance.

Responses of UMass NEAGEP ABD students:

  1. Self-motivation
  2. Perseverance
  3. Having a positive attitude
  4. Knowing how to prioritize (time management)
  5. Resilience
  6. Mental toughness
  7. Grit (courage and resolve)
  8. Previous experience in overcoming challenges
  9. Having something outside the laboratory
  10. Ability to pay attention to detail

Responses of UMass NEAGEP alumni:

  1. Ability to be brave in the face of authority
  2. Perseverance
  3. Coachable with the ability to take negative and positive feedback
  4. Self-motivation
  5. Having something outside the laboratory
  6. Resilience
  7. Mental toughness
  8. Ability to identify resources for support
  9. Knowing how to prioritize (time management)
  10. Endurance

 

Sessions 2 and 3:  Developing a new model of STEM doctoral admissions that does not use GRE test scores

The collected ideas of the workshop participants were synthesized into a new model for selecting STEM doctoral students and is presented below.

A: Pre-Admission Preparation

Increase Applicant Pool:

  1. Identify possible partner minority-serving institutions and invite key faculty from those institutions to an informational workshop.
  2. Host Graduate Preview Weekends for undergraduates from partner institutions.
  3. Develop reciprocal recruiting relationships among institutions (i.e. among NEAGEP institutions and/or GRE Workshop participants).
  4. Advertise that your program does not require GREs.

 

Engage Stakeholders:

  1. Identify administrators who will champion the initiative.
  2. Provide data and seminars to convince skeptics that GREs are not helpful in predicting success in graduate school. Even if not used as cutoffs, GREs may cause collateral damage to students in economically disadvantaged, first generation or underrepresented minority categories because they may be reluctant to apply or may redefine their potential based on their scores.
  3. Facilitate departmental discussions of what qualities they seek in graduate students.
  4. Share Overview of GRE Issues (Appendix 1) with faculty, graduate program directors and university administrators.
  5. Provide admissions committees with diversity and implicit bias training.

 

Develop New Student Assessment Tools:

  1.  Develop a list of departmental/program-specific qualities faculty agree are critical for PhD completion. (See Appendix 1:  Characteristics of Successful PhD Students Identified by Faculty and Administrators (GRE Alternatives Workshop) and Appendix 2: Characteristics that Contribute to Success in Graduate School (Survey of UMass NEAGEP ABD and Alumni).
  2. Determine key courses that are deemed important for success in the program.
  3. Replace letter of recommendation with a Student Assessment developed using the list of qualities identified as important for PhD completion.
    • Ask faculty recommenders to score candidates on the online Student Assessment form and to provide examples justifying score.
    • Ask applicants to score themselves on the Student Assessment form and to provide examples justifying score OR…
    • Provide recommenders and students with the Student Assessment list of qualities to comment upon in the letter of recommendation and in the personal statement; develop a Student Assessment rubric for admissions committee to use in analyzing these materials. AND/OR…
    • Develop problem-solving and/or reading-response questions relevant to graduate program with associated rubrics for scoring them.
    • Develop open response questions to assess valued qualities identified in 1) above. For example, “Describe an obstacle you overcame and…”.  Develop rubric for scoring responses.
    • Provide students with a research article or hypothesis and determine in the interview (below) how many interesting questions they pose.

 B: Implement New Admission Procedure:

  1. Graduate School withholds GRE scores until committee has rank-ordered applicants.
  2. Screen applicants based on whether they have taken the courses deemed critical by the department to prepare them for graduate courses.
  3. Use rubic to score each applicant (two faculty score each applicant): a) on the assessment form (recommender and applicant); b) problem-solving and/or reading-response exercises.
  4. Once Admission Committee screens the applicant pool, engage all faculty who want to recruit graduate students to their research groups in an interview process.
  5. Have two faculty Interview by phone or Skype each applicants deemed qualified in the initial screen; use a standardized interview process to further assess the qualities desired in candidates.
  6. Involve graduate students in on-campus recruitment weekends.
  7. Match each admitted student with near-peer mentor and a faculty mentoring committee; institute measures to ensure that meetings of the mentoring pairs and mentoring committee occur regularly.

 

C: Evaluate New Admissions Procedure:

  1. Prior to changing the admissions procedure, conduct a study in which programs rank order candidates without access to GRE scores. Determine whether order changes when committee knows the GRE scores.
  2. Determine whether new enrollment percentage changes (no. enrollees/no. acceptances).
  3. Determine whether faculty satisfaction with students changes with new procedures.
  4. Work with institutional evaluation professionals to perform long-term study whether diversity and student success change after implementation of the new program.