I CAN Learn^® Algebra I in Central Falls, Rhode Island, 2005-2006
PEGGY C. KIRBY, PH.D.

Intervention

The I Can Learn® education system is a computerized mathematics software solution, specializing in Algebra and pre-Algebra. Aligned to state and national mathematics standards, it presents and assesses content in an interactive learning environment. The teacher in an I Can Learn® classroom is the facilitator of instruction, providing individual, small-group, or whole-class instruction as needed. The 'Classroom Explorer" is a class management/gradebook system that provides real-time feedback on student progress. Teachers use the Classroom Explorer to determine which students need additional assistance with any given concept.

The I Can Learn® system is intended to be used as the primary system of instructional delivery - not as a supplemental or resource tool. The I Can Learn® system consists of 432 lessons from basic mathematics to advanced Algebra concepts. With the help of education service specialists, teachers choose the lessons that align to their local curriculum needs. Each lesson includes a pretest, presentation in a real-world context, guided practice, individual practice, and a posttest. Problem-solving skills also are strengthened with challenging 'journal" problems that require writing in math while solving multi-step application problems. The theoretical foundation for the pedagogical approach comes from Madeline Hunter's seminal work on direct instruction. Hunter (1993, 1995) concluded that effective teachers consistently use eight elements to present lessons. Her research showed these elements to be effective regardless of the teacher's style, student background, grade level, or subject. Complete information about the system is available at www.icanlearn.com

Purpose

This study was intended to serve two purposes. First, it is a comparison of student performance in I Can Learn® Algebra I classrooms versus student performance in traditionally-taught Algebra I classrooms at one school, Central Falls High School. The second purpose was to track performance of students participating in the Rhode Island Department of Labor's demonstration project funded by Governor Carcieri from his statewide Workforce Investment reserve in July 2005.

Population

All students enrolled in Algebra I at Central Falls High School comprised the population targeted for study. In all, there were 18 Algebra I classes; 7 were taught using the I Can Learn® Algebra I software and 11 were taught by traditional means. Six teachers taught Algebra I. One teacher was replaced mid-year. Four of the six teachers taught both I CAN Learn® and traditional classes. The other two teachers taught only traditional classes.

One of the traditionally-taught classes was an honors class for students with high math achievement. One of the I Can Learn® classes was a 'Math Workshop" class for students with very low reading/English language skills. Table 1 shows the total number of students in the population for the remaining 16 classes.

Table 1
Algebra I students at Central Falls High by method of instruction

Course	Method of Instruction	Number of students	Percent of students
Algebra I	I Can Learn® math	127	37.6
Algebra I	Traditional math	177	52.5
Algebra I Honors	Traditional math	13	3.8
Workshop Math	I Can Learn® math	21	6.2
ALL COURSES	% within class type	338	100.0

Population

Because there was no standardized test given in all sections of Algebra I, the Central Falls School District contracted with S. Stuart Flanagan, Professor Emeritus of the College of William and Mary and his colleague, David E. W. Mott, to develop a criterion-referenced test of Algebra I skills. The same 50-item multiple choice test was used as both the pretest, administered in October 2005, and the posttest, administered in June 2006. No validity or reliability data were available for the test which is heretofore referred to as the Benchmark Test.

Sample

Unfortunately, not all teachers administered the Benchmark tests. One teacher forgot to administer the posttest to his class and another teacher failed to administer either test to one of her classes. Both of these were traditionally-taught classes. The numbers of students taking the pretest, posttest, or both are shown in table 2.

Of the 338 students enrolled in Algebra I courses, 156 (46.2%) took both the pretest and posttest. Seventy-three students either added the course after the pretest or dropped the course before the posttest. Mobility and absenteeism, as well as teacher failure to administer the tests, accounted for the low number of students taking both tests. Because students were not randomly assigned to classes, only those students with both scores are considered in the analyses of the effectiveness of the I Can Learn^® Algebra I curriculum. Results are presented for 1) the total sample, 2) the sample without the Workshop math class and Algebra I Honors class, and 4) the Department of Labor demonstration project students.

Table 2
Algebra I students at Central Falls High who took pre- and posttests

Course	Method of Instruction	Number of students who took neither test	Number of students who took pretest only	Number of students who took posttest only	Number of students who took both pretest and posttest
Algebra I	I Can Learn® math	0	45	16	66
Algebra I	Traditional math	17	67	18	75
Algebra I Honors	Traditional math	0	1	2	10
Workshop Math	I Can Learn® math	0	10	6	5
ALL COURSES		17	123	42	156

Analyses

For the 156 students taking both tests, analysis of covariance was used to compare posttest means for I Can Learn^® and traditionally-taught students after adjusting for pretest differences. In the second set of analyses compared means only for students taking a regular Algebra class; students in the honors section and in the Workshop section were excluded. The third set of analyses compares the Department of Labor group to other students.

Results for all students

Students in traditionally-taught classes scored slightly higher on the pretest. Their scores ranged from 1 to 42, with a mean of 17.8. Students assigned to classes that would use I Can Learn^® math scored between 1 and 30 on the pretest, with a mean of 16.5 (see Table 3).

Table 3
Pretest means for traditional and I Can Learn^® math classes

	Instructional method	N	Mean	Standard deviation
Pretest	Traditional	85	17.75	8.09
	I CAN Learn	71	16.46	6.40

Analysis of covariance results showed that students in I Can Learn^® math classes significantly (p=.05) outperformed students in traditional classes at the posttest. I CAN Learn^® students scored about 2 1/2 points higher on the posttest. Table 4 shows the ANCOVA summary table and Table 5 gives the posttest means.

Table 4
ANCOVA summary table for all traditional and I Can Learn^® math classes

Source	Type III Sum of Squares	df	Mean Square	F	p
Corrected Model	4398.97(b)	2	2199.48	37.18	.00
Intercept	4969.24	1	4969.24	84.00	.00
Pretest	4311.89	1	4311.89	72.89	.00
Instructional method	225.84	1	225.84	3.82	.05
Error	9051.11	153	59.16
Total	123638.00	156
Corrected Total	13450.08	155

a Computed using alpha = .05>
b R Squared = .327 (Adjusted R Squared = .318)

Table 5
Posttest means for all traditional and I Can Learn^® math classes

Results for all Algebra students (excludes Honors and Workshop students)

Because the highest-performing students were in the traditionally-taught honors class and the lowest-performing students were in the I CAN Learn^® Workshop class, a more reasonable comparison of the effect of method of instruction would exclude these students. When this second analysis is run, the effectiveness of I CAN Learn^® math is even more pronounced. At the pretest, the two groups have virtually the same mean scores (16.1 for traditional group and 16.7 for the I CAN Learn^® group). In the posttest comparison, however, the difference between groups is 3.4 points (see Tables 6 and 7 and Figure 1).

Table 6
ANCOVA summary table for traditional and I Can Learn^® regular Algebra classes

Source	Type III Sum of Squares	df	Mean Square	F	p
Corrected Model	2684.47(b)	2	1342.23	22.43	.00
Intercept	4956.97	1	4956.97	82.82	.00
Pretest	2194.05	1	2194.05	36.66	.00
Method of instruction	395.41	1	395.409	6.607	.011
Error	8259.29	138	59.850
Total	103370.00	141
Corrected Total	10943.76	140

a Computed using alpha = .05
b R Squared = .245 (Adjusted R Squared = .234)

Table 7
Posttest means for traditional and I Can Learn^® regular Algebra classes

Method of instruction	Mean	Std. Error	95% Confidence Interval
			Lower Bound	Upper Bound
Traditional	24.030(a)	.894	22.263	25.797
I CAN Learn	27.390(a)	.953	25.506	29.274

a Covariates appearing in the model are evaluated at the following values: PRETEST = 16.3617.

Pretest and posttest means for regular Algebra students
by method of instruction
Results for Department of Labor Demonstration Project Students

In compliance with the evaluation requirements for WIA funds, the students in the Rhode Island Department of Labor Demonstration (RIDOL) Project are separately compared to 1) other I CAN Learn^® students and 2) other traditionally-taught Algebra students. These analyses again exclude the honors and workshop classes. No RIDOL students were in traditionally-taught classes.

Pretest means were highest for the RIDOL students (see Table 8); therefore, analysis of covariance was used to adjust for this pretest advantage and compare adjusted posttest scores (see Tables 9 and 10).

Table 8
Pretest means for traditional and I Can Learn^®math classes

	Instructional method	N	Mean	Standard deviation
Pretest	Traditional	75	16.07	6.79
	I CAN Learn non-RIDOL	46	15.78	6.67
	I CAN Learn RIDOL	20	18.80	5.45

Table 9
ANCOVA summary table for traditional, I Can Learn^®non-RIDOL, and I Can Learn^®RIDOL Algebra classes

Source	Type III Sum of Squares	df	Mean Square	F	p
Corrected Model	2703.17(b)	3	901.06	14.98	.00
Intercept	4915.98	1	4915.98	81.73	.00
Pretest	2091.09	1	2091.09	34.76	.00
Class type	414.11	2	207.06	3.44	.04
Error	8240.59	137	60.15
Total	103370.00	141
Corrected Total	10943.759	140

a Computed using alpha = .05
b R Squared = .25 (Adjusted R Squared = .23)

Table 10
Posttest means for traditional, I Can Learn^® non-RIDOL, and I Can Learn^® RIDOL Algebra classes

Class type	Mean	Std. Error	95% Confidence Interval
			Lower Bound	Upper Bound
Traditional	24.03(a	.90	22.26	25.80
I CAN Learn non-RIDOL	27.04(a)	1.15	24.77	29.30
I CAN Learn RIDOL	28.21(a)	1.75	24.75	31.67

a Covariates appearing in the model are evaluated at the following values: PRETEST = 16.36.

Both I CAN Learn^® groups scored significantly higher on the adjusted posttest means than the traditional group. The RIDOL group mean of 28.21was not statistically different than the I CAN Learn^® non-RIDOL group mean of 27.04. The three group means are compared in Figure 2 below.

Discussion

Evidence of the effectiveness of I CAN Learn^® math is apparent in this study. The inclusion of the RIDOL components also yields positive results but not significantly better than I CAN Learn^® alone. Of the Algebra students who took both the pretest and posttest, 19 declined in score or maintained the same score. All but 4 of these were in the traditionally-taught classes. Other students averaged about 11.4 points in improvement, with the RIDOL students gaining the most (an average of 12.3 points).

While this study is promising in its support of the I CAN Learn^® intervention selected at Central Falls High School, three factors support the need for additional study. First, the test used as the dependent measure has not been shown to be content-valid for the Rhode Island curriculum standards. State department administrators worry that it lacks the problem-solving emphasis of the state guidelines. Therefore, a valid test for Rhode Island is needed for additional study. Also, about half of the students were not tested in both Fall 2005 and Spring 2006 and there is no way to know if their progress would have been the same or different than that of the tested students. With such a high level of attrition, a more tightly controlled study is needed. Finally, additional information should be collected about the specific needs and progress of the RIDOL students, both in math classes and in other areas to determine if the intervention is having any carryover effect on attendance, discipline, and achievement in non-math classes.

In spite of these caveats, this preliminary study is very positive and supports the use of I CAN Learn^® math in Rhode Island. It is recommended that the study be repeated during the 2006-2007 school year under controlled testing conditions and using a test created to more closely resemble the NECAP state-mandated criterion referenced tests now given in 8th grade and being piloted this year in 11th grade as the dependent measure. As always, JRL Enterprises, Inc. will provide evaluation support as needed to further our understanding of the effectiveness of its curricula.

References