I CAN Learn^® in Orleans Parish Public Schools
Effects on LEAP 8th Grade Math Achievement, 2003-2004

Peggy C. Kirby, Ph.D.
ed-cet, inc.
October 2006

Students attending Orleans Parish Public Schools were predominantly non-white (93% African American) and from low-income families (77.3% of students receive free or reduced price school lunch)^{This study was conducted in Orleans Parish Public schools before Hurricane Katrina. Demographics in the district today are very different than in 2004.} Orleans was the lowest-performing school district in Louisiana and many of its middle schools, in particular, became eligible for state takeover under the new statewide educational accountability system. Thus, it was an ideal setting for testing the effectiveness of any educational innovation on historically low-performing students.

Participants

Twenty-nine New Orleans public schools serving 8th grade students began using the I CAN Learn^® educational system in mathematics classes just after the second semester of the 2003-2004 school year. At the end of the school year, the school district supplied a data file of all students, including gender, ethnicity, school, teacher, prior year math total score on the Iowa Test of Basic Skills, and current year score on the math portion of the state criterion-referenced test, LEAP. The researcher was allowed to use the file in the Office of Research in June 2004. She first cleaned the file of duplicate names. In many cases, the same students were included two, three, and even four times in the data set. Wherever social security numbers were duplicated, all but the most complete record were dropped. The remaining data set included 6,942 students who had both ITBS and LEAP scores or only one of the two.

Two reasons accounted for the lack of one score or the other. First, the population is highly mobile so many new students entered in 8th grade and many 7th grade students from the prior year did not return. In fact, many Orleans Parish students leave the public school system after elementary and middle school to enter the large private and parochial systems in the area. Second, because many of the 8th grade students were repeating 8th grade, they had no ITBS score from the prior year as ITBS is given only in the 7th grade. LEAP is mandatory and “high-stakes” in the 8th grade. Eighth grade students who fail LEAP must repeat the grade.

With this data set and a data set from the I CAN Learn^® reporting system, the researcher matched files by student and coded the students as participating in either I CAN Learn^® math or traditional (teacher-directed) math. From this merged data set, gender and ethnicity were calculated. Student ethnicity was similar in I CAN Learn^® and traditional classes, but there was a far greater proportion of female students in traditional classes. However, gender data were missing for students who did not have 2004 LEAP scores and some schools did not report any demographics.

The researcher next computed the number of students in each group by school. Because some of these schools used only I CAN Learn^® math, they had no comparison groups. In order to obtain a sample of schools with both traditional and I CAN Learn^® math classes, a decision criterion was set that sample schools had to have at least 20 eighth grade students in traditional math classes and 20 in I CAN Learn^® math classes. Fourteen schools fit this criterion. The district required that the researcher delete all schools not in the sample, as well as student names and social security numbers and teacher names before taking the file for further analysis.

All 14 schools in the sample reported demographics and all used the Pearson SASI Basic Scheduling (3rd edition) software program to randomly assign students to their math classes. Administrators entered the number of necessary classes then allowed the software to create student schedules by randomly filling classes.

According to Tim Hughes, Senior Product Manager, Assessment and Data Analysis Solutions, of Pearson Assessments, SASI Basic Scheduling software uses a random assignment algorithm to schedule students. School clerks supply the number of classes needed and student names and courses needed, then allow the program to schedule students into classes. Because randomization is used, the study incorporates a true experimental design. However, teachers at one magnet school chose after the beginning of the school year to place their lower-performing students in I CAN Learn^® math classes in an effort to improve their test scores. This school was therefore dropped from the sample as it compromised randomization. All students who had 2004 LEAP scores in the remaining 13 schools were included.

13-school sample

The 2,400 students in the 13-school sample were more closely balanced in terms of student gender than in the larger incomplete data set, with 49.6% female students in I CAN Learn^® classes and 51.4% female students in traditional classes (Chi square=.87, p=.38). Both sets had less than ½ % non-minority students. Both had just over 96% African American students and just under 1% Hispanic students.

There were 57 math teachers in the 13 schools. Twenty-nine taught only traditional classes, 15 taught only I CAN Learn^® classes, and 13 taught both.

The amount of time that students used I CAN Learn^® math varied by school, with some students using only the test prep module and others completing up to 95 lessons. The average number of lessons completed was 12.1. Although students are expected to complete about 100 lessons in a full school year, students in this study only used the intervention for less than half the year.

Effects of I CAN Learn^®on LEAP Math Scale Score

The math portion of the 8th grade LEAP exam administered in Spring 2004 was used as the dependent measure to determine the effect of the intervention. Scores of the I CAN Learn^® students were statistically higher (p < .001) than scores of the traditionally taught students. The ANOVA summary table is shown in Table 1 and the means comparison in Table 2. Using the effect size formula, effect size = (Meantreatment - Meancontrol)/SDTotal, the effect size is .35

Table 1
ANOVA Summary Table for Effect of I CAN Learn^® Math on LEAP Math Score

Source	Type III Sum of Squares	df	Mean Square	F	Sig.
Corrected Model	161514.11	1	161514.11	74.50	.000
Intercept	195856783.50	1	195856783.50	90334.46	.000
Class type	161514.11	1	161514.11	74.50	.000
Error	5199173.90	2398	2168.13
Total	202014438.00	2400
Corrected Total	5360688.00	2399

R Squared = .030 (Adjusted R Squared = .030)

Table 2
LEAP Means by Class Type

Class Type	Mean	Std. Deviation	n	95% Confidence Interval
Traditional	278.82	49.82	1318	Lower Bound	Upper Bound
I CAN Learn	295.30	42.26	1082	276.30	281.33
Total	286.25	47.27	2400	292.53	298.08

Figure 1 shows that in all 13 schools, I CAN Learn^® students outperformed traditionally-taught students by as much as 59.5 scale score points on the LEAP math scale score. Means by school for the two groups are presented in Table 3.

Table 3
Mean LEAP scores by school for I Can Learn^® and traditional classes

Effects of I CAN Learn^® on LEAP Math Achievement Level

The LEAP math level is based on the scale score and ranges in 5 steps from Below Basic to Advanced. Students in 2004 were required to pass LEAP at level 2, Approaching Basic, or higher in order to advance to the next grade. I CAN Learn^® students were more likely to meet the LEAP pass criterion and to achieve at the Basic level or higher (see Table 4 and Figure 2).

Table 4
Math Level by Class Type

Effects of I CAN Learn® Math and Teachers on Math Achievement

I CAN Learn^® Mathematics Programs are 100% computer delivered with the teacher as facilitator of instruction. Ten teachers taught in both conditions. Using only those teachers, an analysis of variance was performed using teacher, class type (I CAN Learn^® math or traditional math), and the interaction of teacher and class type as independent variables and math scale score as the dependent variable. Both of the main effects were significant (p<.001), but the interaction effect was not significant (see Table 5.) This finding indicates that the treatment has a positive effect regardless of the teacher; that is, I CAN Learn^® math raised the test scores of all 10 teachers in the study who taught under both conditions. Therefore, it can be concluded that the I CAN Learn^® instruction is functionally free of teacher confound. This meets the letter and spirit of the WWC criterion that allows one teacher per condition when teacher effects are negligible. In this study, all ten teachers’ I CAN Learn^® math means were higher than their traditional class means and the difference was statistically significant (p<.05) for three teachers.

Table 5
ANOVA Summary Table for Effects of Teacher and Treatment on Math Achievement

Notes: Computed using alpha = .05; R Squared = .175 (Adjusted R Squared = .148)

Effects of I CAN Learn® After Accounting for Prior Year Achievement

Although students in the 13 schools were randomly assigned to I CAN Learn^® or traditional classes, prior year math achievement scores were compared as further evidence of pretest equivalence. The 7^th grade ITBS math score was used as a measure of prior achievement. Unfortunately, both 7^th grade ITBS scores and 8^th grade math LEAP scores were not provided for all students. LEAP scores were provided for 2,400 students in the 13 schools. ITBS scores could be matched to 1,360 students (57% of total).

Using the sample of 1,360 students, an analysis of covariance (ANCOVA) was used to determine the effects of I CAN Learn^® on LEAP math scores after adjusting for prior achievement differences. The F of 20.00 was statistically significant (p<.05; see Table 6.) The adjusted LEAP scale score mean for the traditional students was 292.66; for the I CAN Learn students, it was 300.35.

Table 6
ANCOVA Summary Table for Effect of Treatment on Math Achievement After Controlling for Prior Year Achievement

One final note is that of all students in Orleans Parish for whom LEAP scores were provided, the I CAN Learn^® percent passing was 52%.²In just the 13 schools in this study, 40% of traditionally-taught students but 55% of I CAN Learn^® students passed LEAP math. Given these findings, it can be concluded that in the short time it was used in Orleans Parish, the I CAN Learn^® system made a significant contribution to the improvement in high-stakes mathematics test scores in Orleans Parish schools. Positive effects were noted across teachers and schools.