Sources of Evidence

How to Develop a TPP Evaluation

read more

❯

Several aspects or attributes of teacher preparation may not be directly observable but are often of interest to Teacher Preparation Program (TPP) evaluators. These include the quality and substance of instruction, faculty qualifications, effectiveness in preparing new and employable teachers, and success in preparing high-quality teachers.

Evaluations use a variety of evidence, or measures, to estimate the attributes of interest. For example, to gauge how well TPPs prepare high-quality teachers, evaluators could use several different types of evidence, such as performance assessments of teacher candidates, “value-added” estimates of the impact of a particular teacher on the achievement of his or her students, and surveys of employers. The section below lists the most common attributes of interest in TPP evaluations and, for each, the various types of evidence that are typically used to measure them.

INPUT MEASURES

seek to assess the quality of the preparation experience and include such evidence as entrance requirements, course syllabi, and fieldwork policies.

Admissions and recruitment criteria

GPA of incoming class
Average entrance exam scores (e.g., SAT, ACT)
Percentage of minority students in incoming class
Number of candidates in high-need subject areas and specialties

Quality and substance of instruction

Course syllabi
Lectures and assignments
Textbooks
Course offerings and required hours
Required content courses

Quality of student teaching experience

Fieldwork policies, including required hours
Qualifications of fieldwork mentors
Surveys of candidates
Records from observations of student teaching

Faculty qualifications

Percentage of faculty with advanced degrees
Percentage of faculty that are full-time, part-time, adjunct

OUTPUT MEASURES

are used to gauge how well TPPs are preparing new teachers who are employable and effective in the classroom.

Effectiveness in preparing new teachers who are employable and stay in the field

Pass rates on licensure tests
Hiring and retention data

Success in preparing high-quality teachers

Teacher performance assessments administered near end of program
Ratings of graduates by principals/employers
Value-added estimates

Strengths and Limitations
of Commonly Used measures of TPP Quality

Admissions and Recruitment Criteria

Quality and Substance of Instruction

Quality of Student Teaching Experience

Faculty Qualifications

Effectiveness in Preparing Candidates Who Are Employable and Stay in the Field

Success in Preparing High-Quality Teachers

Admissions and Recruitment Criteria

Measure

Average GPA of incoming class

Strength

Single number representing academic ability of the student body
Easy to collect
Easily understood by the general public as an approximation of overall level of incoming students

Limitations

Grading is not uniform across educational institutions
Grades are weak indicators of the quality of training provided by TPP
Average GPA may be less important than the minimum required

Average entrance exam scores

Single number representing academic ability of the student body
Easy to collect
Easily understood by the general public as an approximation of overall level of incoming students

Grading is not uniform across educational institutions
Grades are weak indicators of the quality of training provided by TPP
Average GPA may be less important than the minimum required

Percentage of minority students in incoming class

Encourages TPPs to recruit minority candidates
Easy to collect
Easy to make comparisons across programs
Easily understood by the public

Minority participation rate is a weak indicator of the quality of training provided by TPP
May provide incentive for program to admit students who are academically unprepared and end up dropping out

Number of candidates admitted in high-need areas (e.g., teachers of STEM, special education, English language acquisition)

Encourages TPPs to recruit candidates to teach in high-need areas
Easy to collect
Easy to make comparisons across programs

Distribution of admitted candidates by content area concentration is a weak indicator of the quality of training provided by TPP

Quality and Substance of Instruction

Measure

Course syllabi

Strength

Contract or agreement that a course will cover certain material
Less costly than actually observing courses

Limitations

Syllabi may reflect intended curriculum vs. enacted curriculum (what is actually taught)
Process must be developed and implemented to enable reliable coding of syllabi; can be labor intensive
Syllabi may not reflect instruction—many syllabi are terse, faculty may alter courses mid-stream, and using results for high-stakes decisions may corrupt validity

Lectures and assignments

May be a more accurate reflection than syllabi of what is actually taught

Process must be developed and implemented for reliably coding documents; can be labor intensive
Quantity of documents that needs to be collected and coded makes this costly
Reflect content of instruction, but not quality of instruction

Textbooks

Can give additional information about course coverage

Not all material in the textbook may be covered in the course; other material may be added
Process must be developed and implemented for analyzing textbook content; can be labor intensive

Course offerings and required hours

Easy to collect
Easy to make comparisons across programs

Does not indicate actual quality of instruction in TPP courses

Number of required content courses

Evidence of positive effect on student achievement, especially for secondary mathematics teachers

Courses may not cover content most important for effective K-12 teaching

Quality of Student Teaching Experience

Measure

Fieldwork policies including required hours

Strength

Easy to collect
Easy to make comparisons across programs

Limitations

Does not indicate actual quality of fieldwork experience

Qualifications of fieldwork mentors

One aspect of quality of fieldwork experience

Little empirical evidence links characteristics of mentors to their success in teacher preparation

Surveys of candidates

TPP students can report on actual experience in the field, e.g., frequency of observations, specificity of feedback

Requires development of survey and analysis of responses, which may be time-consuming
Based on individual perceptions; may be biased

Records from observations of student teaching

Can gauge quality of feedback from mentor
Can assess whether candidates are applying what they have learned in the TPP

Requires developing and implementing a method to analyze observation records for TPP evaluation purposes; can be labor intensive

Faculty Qualifications

Measure

Percentage of faculty with advanced degrees, part-time, adjunct, etc.

Strength

Easy to collect
Easy to make comparisons across programs
Face validity—TPP faculty should have appropriate expertise and credentials

Limitations

Many instructors of teacher candidates are in departments other than education and tend not to be included in the evaluation
Little empirical evidence to support connection to effective teacher preparation

Effectiveness in Preparing Candidates Who Are Employable and Stay in the Field

Measure

Pass rates and/or average scores on licensure tests

Strength

Easy to collect

Limitations

Wide variety in tests and cut scores makes comparisons difficult, especially across states
Controversy over rigor and relevance of current exams
Often misinterpreted: indicates that candidates have minimum competencies to enter teaching profession but does not predict future effectiveness in the classroom
May be corrupted (e.g., requiring TPP students to pass a test in order to graduate to ensure 100% pass rates)

Hiring and retention data

Important to potential candidates; face validity

Influenced by numerous geographic and non-TPP factors
May be inaccurate and/or difficult to collect; have to track graduates post-TPP

Success in Preparing High-Quality Teachers

Measure

Teacher performance or portfolio assessments administered near end of program

Strength

Detailed and comprehensive measure of candidates’ skills, results of which can be aggregated to make judgments about TPP outputs
Some evidence shows that these can predict future classroom performance

Limitations

Costly to administer and score
Validity issues arise when candidates can choose what to include in their portfolios

Ratings of graduates by principals/employers

High face validity
Some research shows that principals can accurately identify teachers with low VAM scores

May be costly or time consuming to gather
Subjective; may be biased

Value-added models

Measures teacher impact on student achievement, while attempting to take into account out-of school factors that affect achievement

Requires state to have VAM system in place (not currently the case in most states)
Numerous methodological issues related to reliability and validity still need to be addressed
Incomplete data
Difficult to explain and understand

Sources of Evidence

Purposes of TPP Evaluations

Existing TPP Evaluations

Entities Evaluating TPPs

Sources of Evidence

How to Develop a TPP Evaluation

About the Report

INPUT MEASURES

Admissions and recruitment criteria

Quality and substance of instruction

Quality of student teaching experience

Faculty qualifications

OUTPUT MEASURES

Effectiveness in preparing new teachers who are employable and stay in the field

Success in preparing high-quality teachers

Strengths and Limitations of Commonly Used measures of TPP Quality​

Admissions and Recruitment Criteria

Measure

Average GPA of incoming class

Strength

Limitations

Average entrance exam scores

Percentage of minority students in incoming class

Number of candidates admitted in high-need areas (e.g., teachers of STEM, special education, English language acquisition)

Quality and Substance of Instruction

Measure

Course syllabi

Strength

Limitations

Lectures and assignments

Textbooks

Course offerings and required hours

Number of required content courses

Quality of Student Teaching Experience

Measure

Fieldwork policies including required hours

Strength

Limitations

Qualifications of fieldwork mentors

Surveys of candidates

Records from observations of student teaching

Faculty Qualifications

Measure

Percentage of faculty with advanced degrees, part-time, adjunct, etc.

Strength

Limitations

Effectiveness in Preparing Candidates Who Are Employable and Stay in the Field

Measure

Pass rates and/or average scores on licensure tests

Strength

Limitations

Hiring and retention data

Success in Preparing High-Quality Teachers

Measure

Teacher performance or portfolio assessments administered near end of program

Strength

Limitations

Ratings of graduates by principals/employers

Value-added models

Success! To complete the subscription process, please click the link in the email we just sent you.

Pin It on Pinterest

Strengths and Limitations
of Commonly Used measures of TPP Quality