COMPARISON OF I CAN Learn® AND TRADITIONALLY-TAUGHT 8th GRADE STUDENT
PERFORMANCE ON THE GEORGIA CRITERION-REFERENCED COMPETENCY TEST
PEGGY C. KIRBY, PH.D.
I CAN Learn® is a computerized mathematics software solution, specializing in Algebra and pre-Algebra. Aligned to state and national mathematics standards, it presents and assesses content in an interactive learning environment. The teacher in an I CAN Learn® classroom is the facilitator of instruction, providing individual, small-group, of whole-class instruction as needed. The "Classroom Explorer" is a class management/gradebook system that provides real-time feedback on student progress. Teachers use the Classroom Explorer to determine which students need additional assistance with any given concept.
I CAN Learn® is intended to be used as the primary system of instructional delivery - not as a supplemental or resource tool. I CAN Learn® consists of 303 lessons from basic mathematics to advanced Algebra concepts. With the help of education service specialists, teachers choose the lessons that align to their local curriculum needs. Each lesson includes a pretest, presentation in a real-world context, guided practice, individual practice, and a posttest. Problem-solving skills also are strengthened with challenging "journal" problems that require writing in math while solving multi-step application problems. Complete information about the system is available at www.icanlearn.com.
Gilmer County is a district in northwestern Georgia with a population or 25,000 and annual per capita income of $17,315 (www.gilmerchamber.com/demographics.htm). Gilmer Middle School serves about 900 students in grades 6 through 8. In August 2001, I CAN Learn® was introduced into 8th grade math classes at Gilmer. This evaluation is based on eighth grade students in the 2003-2004 school year who were assigned to either a traditionally-taught or I CAN Learn® classroom based on a stratified random assignment process. Teachers used the students' 7th grade math performance levels to classify students as high, average, or low in math achievement. Once students were stratified based on prior year achievement, they were randomly assigned to I CAN Learn® or traditional classes. Thus, approximately equal proportions of students of each achievement level would be in the treatment and control groups.
Because only one teacher was certified to teach gifted math, she was assigned all gifted students. The I CAN Learn® teacher had only two years teaching experience. She was trained to use the computer classroom the prior year. No students classified to receive special education services were assigned to I CAN Learn® classes. Because all special education and all gifted students were assigned to traditional classes, their scores are not considered in any analyses. Although natural student groups were used, this may also be considered an advantage in terms of external validity in that the setting was not contrived.
The students included in this study consist of all 8th grade students at Gilmer Middle School not classified to receive gifted or other special education services. Tables 1, 2, and 3 describe the demographics of the treatment (I CAN Learn®) and control (traditionally-taught) groups.
Table 1
Student Gender by Class Type
| Gender | Class Type | |||
| Traditional | I Can Learn | Total | ||
| Female | Count | 79 | 50 | 129 |
| % within class type | 48.5% | 54.9% | 50.8% | |
| Male | Count | 84 | 41 | 125 |
| % within class type | 51.5% | 45.1% | 49.2% | |
| Total | Count | 163 | 91 | 254 |
| % within class type | 100.0% | 100.0% | 100.0% | |
| Chi square=0.981; p=0.322 | ||||
Table 2
Ethnicity by Class Type
| Ethnicity | Class Type | |||
| Traditional | I Can Learn | Total | ||
| African Am | Count | 1 | 1 | |
| % within class type | .6% | .4% | ||
| Hispanic | Count | 13 | 5 | 18 |
| % within class type | 8.1% | 5.5% | 7.1% | |
| White | Count | 147 | 86 | 233 |
| % within class type | 91.3% | 94.5% | 92.5% | |
| Count | 161 | 91 | 252 | |
| % within class type | 100.0% | 100.0% | 100.0% | |
| Chi square=2.301; p=0.512 | ||||
Table 3
Lunch Status by Class Type
| Class Type | ||||
| Traditional | I Can Learn | Total | ||
| Free lunch | Count | 61 | 30 | 91 |
| % within class type | 37.4% | 33.0% | 35.8% | |
| Reduced lunch | Count | 21 | 7 | 28 |
| % within class type | 12.9% | 7.7% | 11.0% | |
| Paid lunch | Count | 81 | 54 | 135 |
| % within class type | 49.7% | 59.3% | 53.1% | |
| Total | Count | 163 | 91 | 254 |
| % within class type | 100.0% | 100.0% | 100.0% | |
| Chi square=2.774; p=0.250 | ||||
None of the Chi squares was significant at the p < .05 level, indicating that none of the demographic variables was significantly related to group assignment. Given that the participants were randomly assigned to groups, this finding is as expected.
The GCRCT math test was implemented in Georgia in 2000 in grades 4, 6, and 8. Grades 1, 2, 3, 5, and 7 were added in 2002. The test was designed to assess how well students acquire the skills and knowledge described in the Georgia standards, the Quality Core Curriculum. The math test consists of 60 items in seven sub-parts - Number Sense and Numeration, Geometry and Measurement, Patterns and Relationships/Algebra, Statistics and Probability, Computation and Estimation, and Problem Solving. The Georgia Department of Education offers claims of validity and reliability but does not offer specific technical information (http://www.doe.k12.ga.us/curriculum/testing/crct.asp) Scale scores for the math test range from 150 to 450. Scores that are at or above 350 indicate a level of performance that exceeds the standard for the state test; scores from 300 to 349 indicate a level of performance that meets the standard; and scores below 300 indicate a level of performance that does not meet the standard.
The GCRCT 2004 math scale score and each of the math subscale scores was compared for the I CAN Learn® and traditionally-taught students. Means and standard deviations of the two groups were used to compute effect sizes (see Table 4).
Table 4
Descriptive Statistics and Effect Sizes
| Class Type | N | Mean | Std. Deviation | Std.Error Mean | Effect size * | |
| Total | Control | 163 | 319.8896 | 31.7261 | 2.4850 | .43 |
| ICAN Learn | 91 | 333.5385 | 35.7222 | 3.7447 | ||
| Number Sense | Control | 163 | 325.3129 | 42.6130 | 3.3377 | .45 |
| ICAN Learn | 91 | 344.4945 | 38.6340 | 4.0499 | ||
| Geometry & Measurement | Control | 163 | 321.0920 | 35.7211 | 2.7979 | .17 |
| ICAN Learn | 91 | 327.2857 | 38.0965 | 3.9936 | ||
| Patterns & Relationships | Control | 163 | 323.4785 | 42.5380 | 3.3318 | .33 |
| ICAN Learn | 91 | 337.7033 | 41.9681 | 4.3995 | ||
| Statistics & Probability | Control | 163 | 314.5767 | 32.6874 | 2.5603 | .39 |
| ICAN Learn | 91 | 327.3407 | 36.1255 | 3.7870 | ||
| Computation & Estimation | Control | 163 | 321.4908 | 38.9755 | 3.0528 | .44 |
| ICAN Learn | 91 | 338.7692 | 37.6806 | 3.9500 | ||
| Problem Solving | Control | 163 | 319.6380 | 37.8898 | 2.9678 | .30 |
| ICAN Learn | 91 | 330.9780 | 42.4952 | 4.4547 | ||
| *(treatment mean – control mean)/control SD | ||||||
As shown, all means were higher for the I CAN Learn® group. To determine whether the differences were statistically significant, a .05 probability level was chosen and a multivariate analysis of variance (MANOVA) was used to compare the groups. The decision to use MANOVA was made because of the number of dependent variables being compared. Follow-up t-tests were used after finding a significant MANOVA (Wilk's lambda = .926; F(df = 7) = 2.826; p = .008). The t-test results are presented in Table 5. Note that the assumption of homogeneity of variance was met in all cases (see Levene's test results in Table 5).
Table 5
t-tests of Mean Differences Between I CAN Learn® and Traditional Classses by GCRCT Math Scale
| Levene's Test For Equality of Variances | t-test for Equality of Means | |||||||||
| F | Sig. | t | df | Sig. (2-Tailed) | Mean Difference | Std. Error Difference | 95% Confidence Interval of the Difference | |||
| Equal variances | Lower | Upper | ||||||||
| Total | assumed | .283 | .595 | -3.141 | 252 | .002 | -13.6489 | 4.3456 | -22.2072 | -5.0905 |
| Not assumed | -3.037 | 168.559 | .003 | -13.6489 | 4.4942 | -22.5211 | -4.7767 | |||
| Number sense | assumed | .600 | .439 | -3.555 | 252 | .000 | -19.1816 | 5.3961 | -29.8088 | -8.5544 |
| not assumed | -3.655 | 202.004 | .000 | -19.1816 | 5.2481 | -29.5297 | -8.8336 | |||
| Geometry & Meas | assumed | 1.308 | .254 | -1.294 | 252 | .197 | -6.1937 | 4.7877 | -15.6228 | 3.2354 |
| Not assumed | -1.270 | 176.420 | .206 | -6.1937 | 4.8762 | -15.8168 | 3.4294 | |||
| Patterns& Relation. | assumed | .004 | .947 | -2.568 | 252 | .011 | -14.2248 | 5.5399 | -25.1353 | -3.3143 |
| Not assumed | -2.578 | 188.412 | .011 | -14.2248 | 5.5187 | -25.1112 | -3.3383 | |||
| Statistics & Probab. | assumed | .131 | .718 | -2.873 | 252 | .004 | -12.7640 | 4.4433 | -21.5148 | -4.0132 |
| Not assumed | -2.792 | 171.205 | .006 | -12.7640 | 4.5712 | -21.7872 | -3.7407 | |||
| Computation & Estim. | assumed | .223 | .637 | -3.428 | 252 | .001 | -17.2784 | 5.0404 | -27.2051 | -7.3517 |
| Not assumed | -3.461 | 191.641 | .001 | -17.2784 | 4.9922 | -27.1251 | -7.4317 | |||
| Problem Solving | assumed | .825 | .365 | -2.189 | 252 | .030 | -11.3400 | 5.1815 | -21.5445 | -1.1354 |
| Not assumed | -2.119 | 169.112 | .036 | -11.3400 | 5.3528 | -21.9068 | -.7731 | |||
As shown in Table 5, all mean differences were statistically significant except Geometry and Measurement. Figure 1 graphically depicts the results.
Data were disaggregated by gender, ethnicity, and lunch status to determine the effects of the treatment on important student subgroups.
Gender. Although females scored higher (mean = 329.836) than males (mean = 323.920) overall, the difference was not statistically significant and the treatment was effective for both groups. This conclusion is based on a two-way analysis of variance with gender and class type as the two independent variables. See Tables 6 and 7.
Table 6
Mean Scale Scores by Gender and Class Type
| Mean | Std. Error | 95% Confidence Interval | |||
| Gender | Class Type | Lower Bound | Upper Bound | ||
| Female | Traditional | 327.152 | 3.695 | 319.874 | 334.430 |
| I CAN Learn® | 332.520 | 4.645 | 323.372 | 341.668 | |
| Male | Traditional | 313.060 | 3.584 | 306.001 | 320.118 |
| I CAN Learn® | 334.780 | 5.130 | 324.678 | 344.883 | |
Table 7
ANOVA Summary Table for Effect of Class Type and Gender on GCRCT Math Total
| Source | Type III Sum of Squares | df | Mean Square | F | Sig. |
| Corrected Model | 19079.270 | 3 | 6359.757 | 5.895 | .001 |
| Intercept | 24793433.246 | 1 | 24793433.246 | 22981.875 | .000 |
| GENDER | 2030.268 | 1 | 2030.268 | 1.882 | 171 |
| Class Type | 10642.249 | 1 | 10642.249 | 9.865 | .002 |
| GENDER *Class Type | 3878.224 | 1 | 3878.224 | 3.595 | .059 |
| Error | 269706.384 | 250 | 1078.826 | ||
| Total | 27081148.000 | 254 | |||
| Corrected Total | 288785.654 | 253 | |||
| a R Squared = .066(Adjusted R Squared = .055) | |||||
Class type was statistically significant but gender was not nor was the interaction between the two variables. Thus, I CAN Learn® was effective for both male and female students.
Ethnicity. Because there are few non-white students in Gilmer Middle School, comparisons within ethnic subgroups may not be generalizable to minority students. The mean scale score for white students in the traditional group was 323.00 (SD = 30.52); the meant for I CAN Learn® students was 332.45 (SD = 34.70). The difference was statistically significant (t = 2.168; p = .03). The effect size for white students is .31.
There were only 18 Hispanic students in the sample - 5 in the I CAN Learn® classes and 13 in the traditional classes. Due to the small sample size, caution should be used in interpreting the findings regarding this subgroup. Hispanic students in the I CAN Learn® group scored on average 352.20 (SD = 51.56); Hispanic students in the traditional classes averaged 288.69 (SD = 25.37). A t of 3.56 was statistically significant (p = .003). The effect size is 2.50.
Socioeconomic Status. Using free or reduced price school lunch as a proxy for socioeconomic status (SES), students' scores could be compared by SES subgroup. The usual correlation between SES and student achievement was found. However, students in all three SES subgroups performed better if they were in I CAN Learn® classes. The interaction between lunch status and class type was not statistically significant (F = .264; p = .768).
Table 8
GCRCT Math Total Means by SES and Class Type
| Mean | Std. Error | 95% Confidence Interval | |||
| Class Type | Lunch Status | Lower Bound | Upper Bound | ||
| Traditional | Free | 306.803 | 3.999 | 298.927 | 314.679 |
| Reduced | 317.905 | 6.816 | 304.481 | 331.328 | |
| Student paid | 330.259 | 3.470 | 323.424 | 337.094 | |
| I CAN Learn® | Free | 315.667 | 5.702 | 304.436 | 326.898 |
| Reduced | 324.714 | 11.805 | 301.464 | 347.965 | |
| Student paid | 344.611 | 4.250 | 336.240 | 352.982 | |
Given the findings based on this sample of 254 8th grade students at Gilmer Middle School. It must be concluded that the intervention, I CAN Learn®, significantly improves student achievement in math as measured by the Georgia CRCT. Further, the effects are beneficial in 6 or 7 math subscales, with both ethnic groups tested, with both male and females, and regardless of students' socioeconomic status. Further research is needed to support these findings with additional schools and school districts and to determine the effects on additional ethnic subgroups, particularly African American students.