8 Reliability
Chapter 8 of the Dynamic Learning Maps® (DLM®) Alternate Assessment System 2021–2022 Technical Manual Update—Science (Dynamic Learning Maps Consortium, 2022) describes the methods used to calculate reliability for the DLM assessment system and provides results at five levels, consistent with the levels of reporting. This chapter provides a high-level summary of the methods used to calculate reliability, along with updated evidence from the 2022–2023 administration year.
For a complete description of the methods used to calculate reliability for DLM assessments, including the psychometric background, see the 2021–2022 Technical Manual Update—Science (Dynamic Learning Maps Consortium, 2022) and the corresponding peer-reviewed academic journal articles (Thompson et al., 2019, 2023).
8.1 Background Information on Reliability Methods
The reliability methods and evidence presented in this chapter adhere to guidance given in Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014). Where relevant, evidence provided in accordance with specific standards is noted.
Consistent with the levels at which DLM results are reported, this chapter provides reliability evidence at five levels: (1) the classification accuracy of each linkage level within each Essential Element (EE; linkage level reliability), (2) the number of linkage levels mastered within each EE (EE reliability), (3) the number of linkage levels mastered within each topic in Biology and domain in science grade bands (topic or domain reliability), (4) the total number of linkage levels mastered (subject reliability), and (5) the classification to overall performance level (performance level reliability). As described in the next section, reliability evidence comes from simulated retests in which assessment data are generated for students with the estimated model parameters and student mastery probabilities.
8.2 Methods of Obtaining Reliability Evidence
In creating the simulated retests used to estimate reliability for DLM scores and classifications, we consider the unique design and administration of DLM assessments. Students typically take only 3–5 items per EE. Simulated retests are based on a replication of the administration process, including adaptive routing between testlets, and use students’ known mastery classifications from the operational assessment. Therefore, students may not receive the same testlets in the simulation as they did during their actual assessment (i.e., routing decisions may be different or different testlets may be assigned from the pool of available testlets). This means that the simulated retest offers a genuine approximation of actual retest assignment for any given student. Simulated retests replicate results of DLM assessments from actual examinees, based on administration procedures specific to the DLM assessments. However, the simulation produces approximate estimates of reliability, which are contingent on the accuracy of the current scoring model. That is, reliability estimates are an upper-bound on the true reliability.
Simulated retests were conducted to collect reliability evidence according to the Standards’ assertion that “the general notion of reliability/precision is defined in terms of consistency over replications of the testing procedure” (American Educational Research Association et al., 2014, p. 35). The reliability evidence reported here supports “interpretation for each intended score use,” as Standard 2.0 recommends (American Educational Research Association et al., 2014, p. 42). The “appropriate evidence of reliability/precision” (American Educational Research Association et al., 2014, p. 42) was collected using a methodology that aligns to the design of the assessment and interpretations of results. The procedures used to collect reliability evidence align with all applicable standards.
8.2.1 Reliability Sampling Procedure
The simulation design used to obtain the reliability estimates uses a resampling design to mirror DLM assessment data. In accordance with Standard 2.1, the sampling design uses the entire set of operational assessment data to generate simulated examinees (American Educational Research Association et al., 2014, p. 42). Using this process guarantees that the simulation takes on characteristics of the DLM operational assessment data that are likely to affect reliability results. For a complete description of the simulated retest methodology, see Thompson et al. (2023). For one simulated examinee, the process is as follows:
Draw with replacement the student record of one student from the operational assessment data (i.e., the spring assessment window). Use the student’s original probability scores for linkage level mastery as the true values for the simulated student data.
Simulate new assessment responses based on administration rules See Chapter 4 of the 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017) for a complete description of administration procedures.. Item responses are simulated for the assigned testlets from calibrated model parameters, Calibrated parameters were treated as true and fixed values for the simulated retests. conditional on the linkage level mastery status determined in Step 1.
Score the simulated item responses using the operational DLM scoring procedure, estimating linkage level mastery or nonmastery for the simulated assessment. See Chapter 5 of the 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017) for more information. All three scoring rules were implemented when scoring the simulated retests to be consistent with the operational scoring procedure.
Calculate the aggregated summaries of linkage level mastery for the simulated retests (i.e., the levels of reporting; EE, topic or domain, subject, and performance level).
Compare the estimated linkage level mastery and aggregated summaries from the simulated retests to the values reported for the drawn student on the operational assessment.
Steps 1 through 5 are then repeated 100,000 times for each grade or course to create the full simulated retests data set. Figure 8.1 shows the steps of the simulation process as a flow chart.
Figure 8.1: Simulation Process for Creating Reliability Evidence
Note. LL = linkage level.
8.3 Reliability Evidence
This chapter provides reliability evidence for five levels of data: (1) linkage level reliability, (2) EE reliability, (3) domain and topic reliability, (4) subject reliability, and (5) performance level reliability, which ensures that the simulation and resulting reliability evidence are aligned with Standard 2.2 (American Educational Research Association et al., 2014, p. 42). Additionally, providing reliability evidence for each of the five levels ensures that these reliability estimation procedures meet Standard 2.5 (American Educational Research Association et al., 2014, p. 43).
Reliability evidence at each level is reported using various correlation coefficients. Correlation estimates mirror estimates of reliability from contemporary measures such as the Guttman-Cronbach alpha. For linkage level and conditional evidence by linkage level reliability, the tetrachoric correlation estimates the relationship between true and estimated linkage level mastery statuses. The tetrachoric correlation is a special case of the polychoric correlation in which the variables are dichotomous. Both the polychoric and tetrachoric correlations provide more useful estimates of relationships between ordinal and discrete variables that would otherwise be attenuated using the standard correlation (i.e., the Pearson coefficient). For EE and performance level reliability, the polychoric correlation estimates the relationship between two ordinal variables: the true performance level or true number of linkage levels mastered and the corresponding estimated value. Finally, for subject and topic or domain reliability, the Pearson correlation estimates the relationship between the true and estimated numbers of linkage levels mastered.
Reliability evidence at each level is also reported using correct classification rates (raw and chance corrected), indicating the proportion of estimated classifications that match true classifications. The chance-corrected classification rate, Cohen’s kappa, represents the proportion of error reduced above chance. Kappa values above .6 indicate substantial-to-perfect agreement between estimated and true values (Landis & Koch, 1977). However, Cohen’s kappa may be limited in this context. Numerous studies have shown that the kappa statistic tends to be too conservative when there are unbalanced categories (Cicchetti & Feinstein, 1990; Feinstein & Cicchetti, 1990; O’Leary et al., 2014; Pontius & Millones, 2011). This is often the case for DLM linkage levels, where the base rate of class membership can be less than .3 or greater than .7. See Chapter 5 of this manual for a summary of base rates of class membership. Thus, it is important to interpret the kappa statistic with caution and within the context of the other reporting metrics.
8.3.1 Linkage Level Reliability Evidence
Evidence at the linkage level comes from comparing the true and estimated mastery status for each of the 102 linkage levels in the operational DLM assessment. This level of reliability reporting is the finest grain size of reporting. Although linkage level mastery does not have a directly comparable classical test theory or item response theory analogue, its inclusion is important because it is the level at which mastery classifications are made for DLM assessments. All reported summary statistics of linkage level reliability are based on contingency tables comparing true (operational assessment) and estimated (simulated retest) mastery statuses across all simulated examinees.
In addition to summary statistics from the simulated retests, we also calculate the classification consistency metric, \(\hat{P}_C\), described by Johnson & Sinharay (2018). As the name implies, the classification consistency index is a measure of how consistent the student-level classifications are for each linkage level, and it is calculated from the estimated diagnostic classification model parameters (see Chapter 5 of this manual for a description of the model parameters). The classification consistency metric is based on the estimated model parameters, and thus, it is only applicable to the linkage level, which is the unit of model estimation. See Chapter 5 of the 2021–2022 Technical Manual Update—Science (Dynamic Learning Maps Consortium, 2022) for a complete description of the model specification. This metric is not based on simulated retests, so it provides a measure of reliability independent from the simulated retests.
For each statistic, figures are given comparing the results of all 102 linkage levels. We report linkage level reliability evidence based on three summary statistics from the simulated retests and the nonsimulation-based classification consistency:
- the tetrachoric correlation between estimated and true mastery status,
- the classification agreement for the mastery status of each linkage level,
- the classification consistency (Johnson & Sinharay, 2018), and
- the classification agreement Cohen’s kappa for the mastery status of each linkage level.
Results are given in both tabular and graphical forms. Table 8.1 and Figure 8.2 provide proportions and the number of linkage levels, respectively, that fall within prespecified ranges of values for the four linkage level reliability summary statistics (i.e., tetrachoric correlation, classification agreement rate, classification consistency, and Cohen’s kappa). Table 8.2 provides the four reliability summary statistics for each linkage level.
The correlations and classification agreement rates show reliability evidence for the classification of mastery at the linkage level. Across all linkage levels, 7 (7%) had a tetrachoric correlation below .6, 0 (0%) had a classification agreement rate below .6, 1 (1%) had a classification consistency below .6, and 82 (80%) had a Cohen’s kappa below .6. As previously described, Cohen’s kappa may be limited in this context due to unbalanced class categories. Thus, the other three metrics provide a more useful evaluation of linkage level reliability.
Notably, Johnson & Sinharay (2018) recommend a cutoff of .7 for fair classification consistency. Overall, 84 (82%) linkage levels meet this cutoff, indicating that the linkage level classifications show a high degree of reliability. For a summary of the closely related classification accuracy, see Chapter 5 of this manual.
|
Index range
|
|||||||||
|---|---|---|---|---|---|---|---|---|---|
| Reliability index | 0.00–0.59 | 0.60–0.64 | 0.65–0.69 | 0.70–0.74 | 0.75–0.79 | 0.80–0.84 | 0.85–0.89 | 0.90–0.94 | 0.95–1.00 |
| Tetrachoric correlation | .069 | .059 | .088 | .225 | .176 | .225 | .069 | .069 | .020 |
| Classification agreement rate | .000 | .000 | .000 | .039 | .196 | .314 | .235 | .176 | .039 |
| Classification consistency | .010 | .069 | .098 | .137 | .167 | .225 | .137 | .098 | .059 |
| Cohen’s kappa | .804 | .098 | .029 | .029 | .029 | .010 | .000 | .000 | .000 |
Figure 8.2: Summaries of Linkage Level Reliability
8.3.1.1 Conditional Reliability Evidence by Linkage Level
Traditional assessment programs often report conditional standard errors of measurement to indicate how the precision of measurement differs along the score continuum. The DLM assessment system does not report total or scale score values. Therefore, traditional measures of conditional reliability are not applicable. In particular, standard errors of measurement (inversely related to reliability) that are conditional on a continuous trait are based on the calculation of Fisher’s information, which involves taking the second derivative-model likelihood function with respect to the latent trait. When classifications are the latent traits, however, the likelihood is not a smooth function regarding levels of the trait and therefore cannot be differentiated (Henson & Douglas, 2005; Templin & Bradshaw, 2013). In other words, because diagnostic classification modeling does not produce a total score or scale score, traditional methods of calculating conditional standard errors of measurement are not appropriate. However, because DLM assessments were designed to span the continuum of students’ varying knowledge, skills, and understandings as defined by the three linkage levels, evidence of reliability can be summarized for each linkage level to approximate conditional evidence over all EEs, similar to a conditional standard error of measurement for a total score.
Conditional reliability evidence by linkage level is based on the true and estimated mastery statuses for each linkage level, as summarized by each of the three levels. Results are reported using the same four statistics used to summarize the overall linkage level reliability evidence (i.e., tetrachoric correlation, classification agreement rate, classification consistency, and Cohen’s kappa).
Table 8.3 and Figure 8.3 provide the proportions and the number of linkage levels, respectively, that fall within prespecified ranges of values for each linkage level for the four reliability summary statistics (i.e., tetrachoric correlation, classification agreement rate, classification consistency, and Cohen’s kappa). The correlations and classification agreement rates generally indicate that all three linkage levels provide reliable classifications of student mastery; results are fairly consistent across all linkage levels for each of the four statistics reported. Table 8.2 in the previous section provides the four reliability summary statistics for each individual linkage level.
|
Index range
|
|||||||||
|---|---|---|---|---|---|---|---|---|---|
| Reliability index | 0.00–0.59 | 0.60–0.64 | 0.65–0.69 | 0.70–0.74 | 0.75–0.79 | 0.80–0.84 | 0.85–0.89 | 0.90–0.94 | 0.95–1.00 |
| Initial | |||||||||
| Tetrachoric correlation | .000 | .059 | .029 | .147 | .147 | .471 | .059 | .059 | .029 |
| Classification agreement rate | .000 | .000 | .000 | .088 | .382 | .412 | .088 | .029 | .000 |
| Classification consistency | .000 | .000 | .000 | .059 | .088 | .265 | .324 | .265 | .000 |
| Cohen’s kappa | .588 | .265 | .059 | .029 | .029 | .029 | .000 | .000 | .000 |
| Precursor | |||||||||
| Tetrachoric correlation | .118 | .088 | .176 | .353 | .059 | .118 | .029 | .029 | .029 |
| Classification agreement rate | .000 | .000 | .000 | .029 | .176 | .441 | .235 | .118 | .000 |
| Classification consistency | .029 | .147 | .118 | .147 | .265 | .206 | .059 | .000 | .029 |
| Cohen’s kappa | .941 | .000 | .000 | .029 | .029 | .000 | .000 | .000 | .000 |
| Target | |||||||||
| Tetrachoric correlation | .088 | .029 | .059 | .176 | .324 | .088 | .118 | .118 | .000 |
| Classification agreement rate | .000 | .000 | .000 | .000 | .029 | .088 | .382 | .382 | .118 |
| Classification consistency | .000 | .059 | .176 | .206 | .147 | .206 | .029 | .029 | .147 |
| Cohen’s kappa | .882 | .029 | .029 | .029 | .029 | .000 | .000 | .000 | .000 |
Figure 8.3: Conditional Linkage Level Reliability Evidence Summarized by Linkage Level
8.3.2 Essential Element Reliability Evidence
The first level of linkage level aggregation is the EE. The EE-level results are reported as the highest linkage level mastered for each EE. Because EE-level results are an aggregation of the individual linkage level classifications, more traditional measures of reliability (e.g., the classification consistency used for linkage levels) are not possible. Therefore, reliability results are only reported based on the simulated retests, which do offer a method for evaluating the reliability of aggregated classifications (Thompson et al., 2019).
Three statistics are used to summarize reliability evidence for EEs:
- the polychoric correlation between true and estimated numbers of linkage levels mastered within an EE,
- the classification agreement rate for the number of linkage levels mastered within an EE, and
- the classification agreement Cohen’s kappa for the number of linkage levels mastered within an EE.
Results are given in both tabular and graphical forms. Table 8.4 and Figure 8.4 provide the proportions and the number of EEs, respectively, falling within prespecified ranges of values for the three reliability summary statistics (i.e., classification agreement rate, kappa, correlation). Across all EEs, 3 (9%) had a polychoric correlation below .6, 2 (6%) had a classification agreement rate below .6, and 17 (50%) had a Cohen’s kappa below .6. In general, the reliability summaries provide strong evidence of reliability for the number of linkage levels mastered within EEs. Table 8.5 provides the three reliability summary statistics for each EE.
|
Index range
|
|||||||||
|---|---|---|---|---|---|---|---|---|---|
| Reliability index | 0.00–0.59 | 0.60–0.64 | 0.65–0.69 | 0.70–0.74 | 0.75–0.79 | 0.80–0.84 | 0.85–0.89 | 0.90–0.94 | 0.95–1.00 |
| Polychoric correlation | .088 | .118 | .147 | .441 | .147 | .000 | .029 | .029 | .000 |
| Classification agreement rate | .059 | .265 | .294 | .265 | .059 | .029 | .029 | .000 | .000 |
| Cohen’s kappa | .500 | .353 | .088 | .000 | .029 | .029 | .000 | .000 | .000 |
Figure 8.4: Number of Linkage Levels Mastered Within Essential Element Reliability Summaries
8.3.3 Domain and Topic Reliability Evidence
Science EEs are organized into topics and domains, which are akin to content strands for other assessments. Because Individual Student Score Reports summarize the number and percentage of linkage levels students mastered for each science domain or topic in Biology, For more information, see Chapter 7 of the 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017). we also provide reliability evidence at these levels in their respective grade or course, in accordance with Standard 2.2, which indicates that reliability evidence should be provided consistent with the level(s) of scoring (American Educational Research Association et al., 2014, p. 42).
Reliability at the domain or topic level provides consistency evidence for the number of linkage levels mastered across all EEs in each science domain for each grade or topic for Biology. Domain and topic reliability evidence compares the true and estimated number of linkage levels mastered across all tested levels for each of the three domains and Biology topics. Reliability is reported with three summary statistics:
- the Pearson correlation between the true and estimated number of linkage levels mastered within a domain or Biology topic,
- the classification agreement rate for which linkage levels were mastered as averaged across all simulated students for each domain or Biology topic, and
- the classification agreement Cohen’s kappa for which linkage levels were mastered, as averaged across all simulated students for each domain or Biology topic.
Table 8.6 shows the three summary statistics for each domain in each grade and topic in Biology. Classification rate information is provided in accordance with Standard 2.16 (American Educational Research Association et al., 2014, p. 46). The correlation and Cohen’s kappa summary statistics included in Table 8.6 also align with Standard 2.19 (American Educational Research Association et al., 2014, p. 47). The correlations range from .500 to .942, the average classification agreement rates range from .934 to .995, and the average Cohen’s kappa values range from .840 to .999. These summary statistics indicate that, overall, the domain and topic results provided on score reports are reliable.
8.3.4 Subject Reliability Evidence
The next level of aggregation of linkage level mastery is for the subject overall. Subject reliability provides consistency evidence for the number of linkage levels mastered across all EEs for a given grade or course. Because students are assessed on multiple linkage levels across the assessed EEs in science, subject reliability evidence is similar to reliability evidence for testing programs that use summative assessments to describe overall performance in a subject. That is, the number of linkage levels mastered within a subject is analogous to the number of items answered correctly (i.e., total score) in a different type of testing program.
Subject reliability evidence compares the true and estimated number of linkage levels mastered across all tested levels for a given subject. Because subject-level reporting summarizes the total number of linkage levels a student mastered, the statistics reported for subject reliability are the same as those reported for domain and topic reliability. Reliability is reported with three summary statistics:
- the Pearson correlation between the true and estimated number of linkage levels mastered within a subject,
- the classification agreement rate for which linkage levels were mastered, as averaged across all simulated students, and
- the classification agreement Cohen’s kappa for which linkage levels were mastered, as averaged across all simulated students.
Table 8.7 shows the three summary statistics for each grade and subject. The correlation between true and estimated number of linkage levels mastered ranges from .769 to .948. Students’ average classification agreement rates range from .890 to .971, and average Cohen’s kappa values range from .641 to .896. These values indicate that the total linkage levels mastered in each grade or course are reliably determined.
| Grade | Linkage levels mastered correlation | Average student classification agreement | Average student Cohen’s kappa |
|---|---|---|---|
| 3 | .948 | .971 | .896 |
| 4 | .826 | .928 | .745 |
| 5 | .854 | .933 | .766 |
| 6 | .844 | .929 | .745 |
| 7 | .848 | .922 | .729 |
| 8 | .809 | .890 | .641 |
| 9 | .820 | .911 | .697 |
| 10 | .843 | .931 | .767 |
| 11 | .807 | .909 | .693 |
| 12 | .873 | .926 | .764 |
| Biology | .769 | .913 | .644 |
8.3.5 Performance Level Reliability Evidence
The final level of linkage level mastery aggregation is at the overall performance level. Results for DLM assessments are reported using four performance levels. The scoring procedure sums the linkage levels mastered across all EEs, and cut points are applied to distinguish between the four performance categories. See Chapter 6 of the 2015–2016 Technical Manual—Science (Dynamic Learning Maps Consortium, 2017) and Chapter 6 of the 2018–2019 Technical Manual Update—Science (Dynamic Learning Maps Consortium, 2019) for details on the standard setting procedure to determine the cut points.
Performance level reliability provides evidence for how reliably students are classified to the four performance levels for each subject and grade level. Because the performance level is determined by the total number of linkage levels mastered, large fluctuations in the number of linkage levels mastered, or fluctuation around the cut points, could affect how reliably students are assigned into performance categories. The performance level reliability evidence is based on the observed and estimated performance levels from the simulated retests (i.e., based on the estimated total number of linkage levels mastered and predetermined cut points). Three statistics are included to provide a comprehensive summary of results:
- the polychoric correlation between the true and estimated performance levels within a grade and subject,
- the classification agreement rate between the true and estimated performance levels within a grade and subject, and
- the classification agreement Cohen’s kappa between the true and estimated performance levels within a grade and subject.
Table 8.8 presents this information across all grades in science. Polychoric correlations between true and estimated performance levels range from .798 to .972. Classification agreement rates range from .662 to .848, and Cohen’s kappa values are between .620 and .911. These results indicate that the DLM scoring procedure of reporting performance levels based on total linkage levels mastered results in reliable classification of students to performance level categories.
| Grade | Polychoric correlation | Classification agreement rate | Cohen’s kappa |
|---|---|---|---|
| 3 | .972 | .848 | .911 |
| 4 | .859 | .743 | .709 |
| 5 | .886 | .789 | .727 |
| 6 | .876 | .753 | .762 |
| 7 | .877 | .727 | .764 |
| 8 | .824 | .679 | .689 |
| 9 | .838 | .662 | .689 |
| 10 | .860 | .723 | .734 |
| 11 | .828 | .672 | .685 |
| 12 | .897 | .707 | .760 |
| Biology | .798 | .712 | .620 |
8.4 Conclusion
In summary, reliability measures for the DLM assessment system address the standards set forth by American Educational Research Association et al. (2014). The methods are consistent with assumptions of diagnostic classification modeling and yield evidence to support the argument for internal consistency of the program for each level of reporting. The results indicate high levels of reliability for the individual linkage level mastery classifications, as well as for all levels of aggregation for which results are reported (i.e., EE, domain and topic, subject, and overall performance level). Because the reliability results depend on the model used to calibrate and score the assessment, any changes to the model or evidence obtained when evaluating model fit also affect reliability results. As with any selected methodology for evaluating reliability, the current results assume that the model and model parameters used to score DLM assessments are correct. However, unlike other traditional measures of reliability that often require unrealistic assumptions about equivalent test forms, the simulation method described in this chapter provides a replication of the same test administration process used in the operational assessment, which provides a rigorous evaluation of the variation in student results across simulated repeated assessment administrations.