Measuring Teachers’ Use of
Standards-based Instructional
Materials
Karen D. King
This material is based upon work supported by the National Science Foundation under Grant No.
DRL-0732184. Any opinions, findings, and conclusions or recommendations expressed in this
material are those of the authors and do not necessarily reflect the views of the National Science
Foundation
Overview
Our framing of the problem - measuring
textbook use
 Research question
 Data collection
 Results
 Interpretation
 Implications for practice and future
research
 Other areas of research

Instructional Materials vs. Curriculum
•
Instructional materials are not curriculum
• Curriculum is a plan or pathway envisioned from
students' current understanding to target
understandings as outlined in Standards
• Instructional materials are tools and resources
comprised of a series of instructional tasks and
exposition teachers can use to teach the
curriculum
•
Often, without curriculum guidance, the
textbook/instructional materials become the
curriculum, substituting instructional
materials designers' goals and plans for that
of the district or teacher
Why does this distinction matter?

A major concern in implementation is
alignment with often conflicting policies
◦ State Standards
◦ District Curriculum
◦ District/School Textbook/Instructional
Materials Adoption
◦ State Assessments
Our Perspective on Measuring
Implementation
Considers the alignment of textbook to
curriculum (Mitchell, King & Gearhardt,
in revision)
• Considers alignment of curriculum to
state standards (Mitchell, et al., in revision;
Porter, 2002)
• Considers alignment of textbook,
curriculum, and state standards to state
achievement tests (King, et al., in revision)
•
An Example
Fig 3. Grade 8 Cognitive Demand
0.00
0.20
0.40
Memorize/Recall
Perform Procedures
Demonstrate Understanding
Conjecture, Generalize, Prove
Solve non-routine problems/ Make Connections
NJ Stnds 2009 Gr. 8
CMP2 All Units Gr.8
0.60
0.80
Research Question
• What
is the relationship between
middle grades mathematics teachers'
use and adaptation of Standardsbased instructional materials and
students' achievement?
Research Setting
Newark, NJ in 2008-2009 school year
• Had adopted Connected Mathematics
Project (CMP) in 2002, started using CMP
2 in 2008-2009
• Had an NSF-funded Local Systemic
Change project that provided substantial
professional development on CMP and
middle grades mathematics education
through 2008
•
Data Collection
Part of a larger mixed methods study
• Surveyed 159 middle grades mathematics
teachers and the student achievement data
from 2528 of their students in their first
mathematics class of the week
• In Spring 2009, teachers completed the Surveys
of Enacted Curriculum Instructional Content
and Instructional Practices surveys and projectdeveloped CMP
• Students took the NJ ASK during Spring 2009
•
Surveys of Enacted Curriculum
Demographic information on the teacher,
teacher qualifications, and professional
development experiences
 Information on the teachers’ instructional
practices
 Information on the content and cognitive
demand of the instruction
 To ensure accurate responses, we conducted
a professional development session on
assessing the cognitive demand of tasks for
students prior to teachers completing the
survey

SEC Instructional Content Survey
Time on Topic For 188 content areas ranging from
number to advanced algebra, the SEC-IC asks what
proportion of time is spent teaching the topic
 Emphasis For each content area that the teacher
indicates he or she teaches, the SEC-IC asks what
proportion of time spent teaching the topic was
spent at 5 levels of cognitive demand

◦ Memorize facts, perform procedures, demonstrate
understanding, conjecture/generalize/prove, solve nonroutine problems/make connections

Prior to survey administration, teachers participated
in professional development to ensure that the
meanings for each was established and common
CMP Implementation Survey
Background information on teachers’
experiences with CMP
 General questions about the teachers’
use of other materials and context of
teaching
 For each lesson organized by units, we
asked teachers to describe their use of
each lesson in the textbook with the
following instruction:

To answer the following questions, use the
following definitions of textbook use

Use without modification – Follows the outline of the lesson as described in
the teachers’ materials. No changes are made to the task and the structure of
the lesson follows the Launch, Explore, Summarize routine described in the CMP
materials. The goals and objectives of the lesson remain the same.

Use with adaptation – Follows the outline of the lesson as described in the
teachers’ materials with changes made to the task, but not the structure of the
lesson. The goals and objectives of the lesson remain the same. The structure of
the lesson follows the Launch, Explore, Summarize routine described in the CMP
materials, but the task is modified in one of several ways

Use as a one of many resources – Takes tasks from the materials but uses
them to create own lesson with own lesson structure.

Replace – Does not use the materials but takes tasks or lessons from other
materials to teach the goals and objectives of the lesson.

Does not use – Does not use any materials to teach the goals and objectives of
the lesson.
Alignment of NJ ASK to New Jersey
Mathematics Standards and
Instructional Materials
NJASK
2009
State Mathematics
Standards
Grade 6
Grade 7
Grade 8
.29
.30
.30
Connected Mathematics 2
Instructional Materials
.21*
.26
.25
Alignment considers both the content
and cognitive demand alignment (see
Porter, et al., 2008)
Data Analysis
•
•
Includes 143 teachers and their 2107
students
Removes teachers we did not have
information on:
• their position type (general or special education),
• teachers whose classrooms were comprised of
very mobile students
• students for whom we did not have 2008 math
scores
•
About 1/3 of teachers in the sample were
special education teachers
Measure construction
•
To measure use of the materials, we created the
curriculum coverage index (CCI) and to
measure adaptation, we created the lesson
modification index (LMI).
• CCI is a simple proportion of the CMP lessons that
were not taught during the year
• LMI represents the extent to which the teacher has
modified the lesson, expressed on a three-point scale
where the low score represents all lessons as used
without adaptation and the high score represents all
lessons used as one of many resources.
Measure Construction

The SEC creates cognitive demand measures
that sum to 1 based on emphasis of each of
the types of cognitive demand


Memorize facts, perform procedures, demonstrate
understanding, conjecture/generalize/prove, solve nonroutine problems/make connections
Using factor analysis on these SEC measures,
we created a new cognitive demand measure
which is characterized by stressing
"procedures" and avoiding "non-routine
problems"
Results: Incomplete table ofHLM Fixed Effects Estimates for a Model of
2009 NJ ASK Mathematics Scale Score with Robust Errors
Fixed Effect
Model
Intercept
203.54 (2.41)
Level 1
2008 Math Score 0.72** (0.03)
Special Education -14.52** (2.39)
LEP -0.73 (3.27)
Black -3.76 (2.43)
Latino -0.96 (2.02)
Level 2
LMI 4.191* (2.06)
CCI -0.26**( 0.07)
Cognitive Demand Factor 2.82* (1.28)
Summary of Results






Teachers’ increased use of the CMP instructional materials as
compared to average is significantly related to higher student
achievement.
However, in relationship to traditional measures of fidelity, where the
focus is on use without adaptation, our findings suggest that greater
than average adaptation on the part of the teacher is related to
increased achievement of students.
A focus on procedures and decreased emphasis on non-routine
problems is significantly related to higher student achievement.
Instructional Practice scales were not statistically significant.
Teachers’ alignment was not statistically significant association with
student achievement.
Also, while 2008 achievement and special education classifications of
students remained significant factors in predicting 2009 student
achievement, race/ethnicity and LEP status did not.
O’Donnell’s 5 Criteria for Measuring
FOI





Adherence—whether the components of the intervention
are being delivered as designed;
Duration—the number, length, or frequency of sessions
implemented;
Quality of delivery— the manner in which the
implementer delivers the program using the techniques,
processes, or methods prescribed;
Participant responsiveness—the extent to which
participants are engaged by and involved in the activities
and content of the program; and
Program differentiation—whether critical features that
distinguish the program from the comparison condition
are present or absent during implementation. (O’Donnell,
2008, p. 34)
Our Study
O’Donnell’s Criteria

Curriculum Coverage (CCI)

Extent of use or adherence

Alignment (includes time and
cognitive demand emphasis

Duration

Quality of Delivery

adaptation to local contexts
of the materials (LMI)

Decreased cognitive demand
Difference in measures of FOI
Limitations

We do not account for teachers’ mathematical
knowledge for teaching in our model.
◦ The SEC asks questions about certifications and degrees, but,
there was little variability in teacher qualifications of this type
 Only 6 teachers (3.8%) had secondary mathematics certification
 Only 11 teachers (7.1%) had a major in mathematics or mathematics
education at either the undergraduate or Masters level.

The lack of fine-grained data collection for teachers’
years of experience, especially in the early years when
student learning gains are most pronounced (Clotfelter et
al., 2007).
◦ However, given our results for years teaching with the CMP
instructional materials, we may have produced the same general
results for teaching experience with a better measure.
Limitations Continued
Advanced
Proficient
Proficient
Partially
Proficient
Returning to the Example
Fig. 2 Grade 7 Cognitive Demand
0.00
0.20
0.40
0.60
0.80
Memorize/Recall
Perform Procedures
Demonstrate Understanding
Conjecture, Generalize, Prove
Solve non-routine problems/ Make Connections
NJ Stnds 2009 Gr. 7
CMP2 All Units Gr.7
Fig 3. Grade 8 Cognitive Demand
0.00
0.20
0.40
Memorize/Recall
Perform Procedures
Demonstrate Understanding
Conjecture, Generalize, Prove
Solve non-routine problems/ Make Connections
NJ Stnds 2009 Gr. 8
CMP2 All Units Gr.8
0.60
0.80
NJ ASK Item Types by Grade
Item
Count
by
Grade 6
Grade 7
Grade 8
MC
42
42
42
SCR
8 (noncalculator)
10
10
ECR
5
5
5
5
5
5
Type
# of sections
Total raw score
points possible
50
52
52
120 min.
124 min.
133 min.
(excluding field test items)
Approximate total testing
time
MC - multiple
choice, 1 raw
score point
SCR - short
constructed
response, 1
raw score
point
ECR –
extended
constructed
response, 3
raw score
points
Implications for Research in an Era
of CCSSM
To the extent there was lack of alignment between the NJ
ASK, the standards, and the instructional materials, it was to
decrease cognitive demand as opposed to content coverage.
 How well the outcome measure aligns with the standards,
curriculum and the instructional materials is a critical factor
in the results of such studies.
 How well the PARCC and SBAC assessments will align with
the CCSSM, particularly the Standards for Mathematical
Practice will have implications for cross-state research on
instructional materials use
 In an era of openness and the Internet, we need to better
understand what teachers use outside the adopted
instructional materials, and how this use supports or
detracts from a cohesive and coherent curriculum

Suggestions for Policy and Practice


As districts adopt new instructional
materials and regimes, implementation
strategies and monitoring should focus on
both fidelity to structure and process.
As textbook authors design and modify
texts, they should consider guidance to
districts and teachers that support
implementation by providing sufficient
description of both the structure and the
process of implementation that allow them
to make appropriate adaptations.