Module 3: Measurement and Data Capacity

Resources & Perspectives

Now that you have recorded your initial thoughts to the challenge questions, take some time exploring the resources below.

Four types of resources are provided:
Documents to read, Audio files to listen to, Videos to watch, and links to other sites.

After you have referred to these resources, click the 'next' button below to navigate to Revised Thoughts in order to consider again the questions posed in the Challenge Scenario.

---

Do you think attainment-oriented measures are adequate for assessing teacher and school effectiveness, or are growth and value-added measures more accurate?

The utility of attainment and value-added systems for measuring teacher and school effectiveness (4:07)
by Robert Mendro
Assistant Superintendent for Research and Evaluation at Dallas Public Schools

Utility of attainment and value-added measures (1:50)
by Dash Weerasinghe
Director of Institutional Research, Dallas Independent School District

Adequate assessment measures (1:44)
by Brad Carl
Embedded Researcher in Milwaukee Public School System, University of Wisconsin-Madison

Value-added versus growth (0:47)
by Rob Meyer
Senior Scientist and Director, Value-added Research Center, University of Wisconsin-Madison

Attainment measures are inherently biased (1:39)
by Deb Lindsey
Director of Research and Assessment, Milwaukee Public Schools

Broad benefits of utilizing value-added measures (1:54)
by Karen Soehnge
Chief Academic Officer, Houston ISD

Value-added challenges in small districts (2:53)
by Lee Holcomb
Director of Texas Schools Project

Value-added considerations for small districts (2:18)
by Daniel O’Brien
Texas Schools Project, The University of Texas at Dallas

Solutions for value-added in small districts (1:28)
by Daniel O’Brien
Texas Schools Project, The University of Texas at Dallas

Value-added in small districts (2:38)
by Rob Meyer
Senior Scientist and Director, Value-added Research Center, University of Wisconsin-Madison

Attainment versus value-added (4:01)
by Chris Thorn
Assistant Scientist, Value-added Research Center, University of Wisconsin-Madison

From attainment to value-added measures (2:12)
by Daniel O’Brien
Texas Schools Project, The University of Texas at Dallas

The ability of value-added to control for factors beyond teacher effects (2:18)
by Doug Harris
University of Wisconsin-Madison

Scrapbooks instead of snapshots (1:15)
by Bruce Gearing
Assistant Superintendent of Curriculum & Instruction, Mt. Pleasant Independent School District

What is value-added (0:26)
by Mike Christian
Assistant Scientist, Wisconsin Center for Education Research, University of Wisconsin-Madison

Are attainment measures adequate for assessing student performance? (1:20)
by Lee Holcomb
Director of Texas Schools Project

Growth and value-added in Dallas (3:37)
by Dash Weerasinghe
Director of Institutional Research, Dallas Independent School District

How regional educational service centers might engage small districts regarding value-added (1:47)
by Barron White
Field Service Agent-Region 19

Measurement considerations (0:46)
by Rob Meyer
Senior Scientist and Director, Value-added Research Center, University of Wisconsin-Madison

Capacity considerations when moving from attainment to value-added modeling (1:19)
by Doug Harris
University of Wisconsin-Madison

Issues associated with value-added in small districts (4:22)
by Chris Thorn
Assistant Scientist, Value-added Research Center, University of Wisconsin-Madison

More Resources (show/hide)

How are you going to measure performance at the early grades, in the upper grades, and in subjects without standardized tests?

Inclusion of non-tested teachers (2:33)
by Tony Milanowski
Assistant Scientist, Wisconsin Center for Education Research, University of Wisconsin-Madison

The central importance of high quality assessments for teachers in all grades and subject areas (3:07)
by David Lussier
Special Assistant to the Superintendent, Austin ISD

Including teacher of non-tested subjects and grades in the measurement and performance award system (2:04)
by Robert Mendro
Assistant Superintendent for Research and Evaluation at Dallas Public Schools

The inclusion of non-tested grades and subject areas in performance award programs (2:27)
by Lee Holcomb
Director of Texas Schools Project

Explanation of rubric used for student learning objectives (3:06)
by David Lussier
Special Assistant to the Superintendent, Austin ISD Referenced rubric guide

Considerations for including a wider array of grade and subject area teachers in performance award systems in small districts (6:01)
by Chris Thorn
Assistant Scientist, Value-added Research Center, University of Wisconsin-Madison

Broadening the net (3:01)
by Sam Zigrossi
Program Director, The Charles A. Dana Center

Including non-tested grade and subject areas (1:11)
by Bruce Gearing
Assistant Superintendent of Curriculum/Instruction, Mt. Pleasant Independent School District

Assessing all teachers (3:00)
by Deb Lindsey
Director of Research and Assessment, Milwaukee Public Schools

Challenges to and solutions for including non-tested subjects and grades in performance award systems (9:37)
by Chris Thorn
Assistant Scientist, Value-added Research Center, University of Wisconsin-Madison

Value-added in upper and early grades (2:16)
by Brad Carl
Embedded Researcher in Milwaukee Public School System, University of Wisconsin-Madison

Assessment quality is critical (2:13)
by Steve Midgeff
Unknown

More Resources (show/hide)

How do we achieve balance in the measurement system so that the things we value get taught in the ways we want them to be taught?

Tensions inherent in high stakes accountability systems (3:40)
by David Lussier
Special Assistant to the Superintendent Austin ISD

Multiple measures are necessary (0:50)
by Sam Zigrossi
Program Director, The Charles A. Dana Center

The importance of including multiple measures of student performance (2:14)
by Bruce Gearing
Assistant Superintendent of Curriculum & Instruction, Mt. Pleasant Independent School District

Balancing tests with additional indicators related to student, teacher and school goals (2:54)
by Lee Holcomb
Director of Texas Schools Project

Ways to achieve balance in the measurement system (3:30)
by Chris Thorn
Assistant Scientist, Value-added Research Center, University of Wisconsin-Madison

Correlation between teacher observation data and value-added scores (1:45)
by Robert Mendro
Assistant Superintendent for Research and Evaluation at Dallas Public Schools

The use of multiple assessments in the value-added measurement system (3:04)
by Dash Weerasinghe
Director of Institutional Research, Dallas Independent School District

Balance in the measurement system: Student learning objectives and assessments (9:15)
by David Lussier
Special Assistant to the Superintendent Austin ISD

Including grades 10 - 12 in the performance award system (2:24)
by Steve Midgeff
Unknown

The inclusion of resource teachers in the performance award system (2:27)
by Steve Midgeff
Unknown

More Resources (show/hide)

How can a district ensure a high level of data quality?

Data management and quality considerations -OR- Program monitoring and data management (29:02)
by Robert Mendro
Assistant Superintendent for Research and Evaluation at Dallas Public Schools Referenced Presentation

Ways to ensure data quality in performance award systems (5:05)
by Chris Thorn
Assistant Scientist, Value-added Research Center, University of Wisconsin-Madison

Highest quality data (1:53)
by Bruce Gearing
Assistant Superintendent of Curriculum/Instruction, Mt. Pleasant Independent School District

The importance of accurate student–teacher linkages to ensuring data quality (2:42)
by Daniel O’Brien
Texas Schools Project, The University of Texas at Dallas

Data quality (5:10)
by Deb Lindsey
Director of Research and Assessment, Milwaukee Public Schools

Ensuring high levels of data quality (1:13)
by Robert Mendro
Assistant Superintendent for Research and Evaluation at Dallas Public Schools

District level data capacity challenges (3:52)
by Lee Holcomb
Director of Texas Schools Project

Ensuring data quality at payout time (1:27)
by David Lussier
Special Assistant to the Superintendent Austin ISD

Ensuring data quality (1:13)
by Brad Carl
Embedded Researcher in Milwaukee Public School System, University of Wisconsin-Madison

Data capacity issues associated with scale-up (3:38)
by David Lussier
Special Assistant to the Superintendent Austin ISD

Data required for value-added (2:06)
by Bob Glover
Sr. Information Process Consultant, Wisconsin Center for Education Research, University of Wisconsin-Madison

Data quality issues (0:43)
by Sara Kraemer
Assistant Scientist, Wisconsin Center for Education Research, University of Wisconsin-Madison

Data system needs (1:50)
by Drew Scheberle
Vice President, Education & Workforce Development

Management of the system (1:16)
by Steve Midgeff
Unknown

The importance of teamwork and inter-departmental coordination (1:44)
by Steve Midgeff
Unknown

More Resources (show/hide)

How can a district communicate clearly to all constituent groups regarding its measures of teacher and school effectiveness and the determination of awards?

Communicating about value-added to key stakeholders (1:45)
by Robert Mendro
Assistant Superintendent for Research and Evaluation at Dallas Public Schools

Communication essential and support is beneficial, especially surrounding the payout process (4:14)
by Carol Ruckel
Director, Weld County Teacher Incentive Fund Project

Communication at payout time (5:05)
by David Lussier
Special Assistant to the Superintendent Austin ISD

Communicating the value-added model to teachers (1:25)
by Steve Midgeff
Unknown

Best practices in domain of communication (1:24)
by Brad Carl
Embedded Researcher in Milwaukee Public School System, University of Wisconsin-Madison

Communicating to teachers and stakeholders (1:31)
by Bruce Gearing
Assistant Superintendent of Curriculum/Instruction, Mt. Pleasant Independent School District

Ongoing collaboration is essential (0:49)
by Steve Midgeff
Unknown

More Resources (show/hide)

General insights regarding measurement and performance award systems

Leadership as critical component for incorporating value-added measurement into district practice (1:29)
by Robert Mendro
Assistant Superintendent for Research and Evaluation at Dallas Public Schools

The importance of principal leadership to the success of performance award programs (1:05)
by Sheila Anderson
Director of Administrative Programs, Strategic Compensation for Austin ISD

Advice regarding the use of value-added beyond the performance award system (1:03)
by Deb Lindsey
Director of Research and Assessment, Milwaukee Public Schools

Value-added challenges in small and large districts (1:54)
by Robert Mendro
Assistant Superintendent for Research and Evaluation at Dallas Public Schools

Key elements of the Dallas value-added measurement system (2:10)
by Dash Weerasinghe
Director of Institutional Research, Dallas Independent School District

The idea of value-added is to try to account for those other factors (2:18)
by Doug Harris
University of Wisconsin-Madison

Resources for creating a value-added model (1:01)
by Dash Weerasinghe
Director of Institutional Research, Dallas Independent School District

Is the pain worth the potential gain? (0:57)
by Sheila Anderson
Director of Administrative Programs, Strategic Compensation for Austin ISD

Access to the student management system (7:43)
by Steve Midgeff
Unknown

The construction of comparison groups (2:34)
by Steve Midgeff
Unknown

UNKNOWN TITLE (6:05)
by Dale Ballou
Unknown

UNKNOWN TITLE (1:19)
by Doug Harris
University of Wisconsin-Madison

Teacher quality as a system of different parts (2:19)
by Doug Harris
University of Wisconsin-Madison

More Resources (show/hide)

Current Research on Measurement and Value-Added

The following section of resources is a collection of papers and presentations made at the National Conference on Value-Added Modeling at the University of Wisconsin at Madison in April, 2008. While technical in nature, the papers provide an array of perspectives on the state-of-the-art in value-added modeling. For several of the papers, author interviews provide synopses of key points and discuss the implications of the research for performance award systems.

The national conference on value-added by Doug Harris (UW-Madison) Presentation

Test scaling and value-added measurement by Dale Ballou (Vanderbilt University) Paper, Presentation

Value-added assessment in practice: Lessons from the pennsylvania value-added assessment system pilot project by Dan McCaffrey (RAND), & Laura Hamilton (RAND) Summary, Paper, Presentation

Vertical scaling in value-added models for student learning by Derek Briggs (UC-Boulder), Ed Wiley (UC-Boulder), & Jonathan Weeks (UC-Boulder) Summary, Paper, Presentation, Interview (2:36)
Discussion by Henry Braun (Boston College)

The sensitivity of value-added modeling to the creation of a vertical score scale by Derek Briggs (UC-Boulder), Ed Wiley (UC-Boulder), & Jonathan Weeks (UC-Boulder) Paper

Measuring effect sizes, the effect of measurement error by Donald Boyd (SUNY at Albany), Hamilton Lankford (SUNY at Albany), Susanna Loeb (Stanford University), James Wyckoff (University of Virginia), & Pam Gorssman (Stanford University) Summary, Paper, Presentation
Discussion by Brian Jacob (University of Michigan)

Would accountability based on teacher value-added be smart policy? by Doug Harris (UW-Madison) Paper

The policy uses and 'policy validity' of value-added and other teacher quality measures by Doug Harris (UW-Madison) Summary, Paper, Presentation, Interview (2:19)

Value-added in english schools by Helen Evans (England’s National Department for Children, Schools and Families) Summary, Paper, Presentation

Some econometric considerations for value-added modeling by Jeffrey Wooldridge (Michigan State University) Presentation

Student sorting and bias in value-added estimation: selection on observables and unobservables by Jesse Rothstein (Princeton University) Summary, Paper, Interview (4:10)
Discussion by Jane Cooley (UW-Madison)

Exploring student-teacher interactions in longitudinal achievement data by JR Lockwood (RAND), & Dan McCaffrey (RAND) Summary, Paper, Presentation
Discussion by Julian Betts (UC-San Diego)

The intertemporal stability of teacher effects estimates by JR Lockwood (RAND), Dan McCaffrey (RAND), & Tim Sass (Florida State University) Summary, Paper, Presentation, Interview (6:43)

A generalized value-added model with conditional random effects and multivariate shrinkage by Robert H. Meyer (UW-Madison) Presentation
Discussion by Chris Taber (UW-Madison)

Decomposition of estimates in a layered value-added assessment model by S. Paul Wright (SAS EVAAS), & Bill Sanders (SAS EVAAS) Paper

Adaptive centering with random effects: An alternative to the fixed effects model for time-varying treatments by Steve Raudenbush (University of Chicago) Summary, Paper, Presentation

Assumptions of value-added models for estimating school effects by Steve Raudenbush (University of Chicago), & Sean Reardon (Stanford University) Summary, Paper, Presentation
Discussion by Joseph Martineau (Michigan Department of Education)

Impediments to the estimation of teacher value-added by Steve Rivkin (Amherst College), & Jun Ishii (Amherst College) Paper, Presentation
Discussion by Tom Smith (Vanderbilt University)

Are teacher-level value-added estimates biased: An experimental validation of non-experimental estimates by Thomas Kane (Harvard University), & Douglas Staiger (Dartmouth College) Summary, Paper, Presentation