After a year of delay, the final report on the United States Department of Education study on educational software is out. Flying in the face of 20 years of studies and metastudies that show the effective adoption of eLearningi [0] will increase student academic performancei [0] by over one letter grade, this study reports: “Test Scores Were Not Significantly Higher in Classrooms Using Selected Reading and Mathematics Software Products.” Page 1 of the Executive Summary.

Mathematica Policy Researchi [0] Inc. with SRI International was engaged to design and conduct the study. The 140 page, March 2007, “Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort, Report to Congress” is on the MPR web site:

http://www.mathematica-mpr.com/publications/PDFs/effectread.pdf [1]

As reported by the Washington Post:

http://www.msnbc.msn.com/id/17956093/from/ET/ [2] )

The $12 million study evaluated 15 reading and math products used by 9,424 students in 132 schools in the 2004-2005 school year. Assuming that each teacher taught one class of children then an estimated 439 teachers were charged with transforming their classroom and practice to use these reading and math products.

I have read the 2003 Issues and Recommendations report, 2004 Statement of Work, and now the MPR/SRI report. Over the years I have also talked to several of the major players in the study, and read many of the studies referenced in their report.

This is my take on this study when referenced to what eSATSi [2] has determined as an effective means to implement digital curriculum in a classroom.

1. The program assessment and management costs were about $3 million, leaving about $9 million for direct costs of the study.

1. The implementations used hardware and systems technology available in the school. Effective eLearning requires continuous student accessibility to a computer interface. There was no data presented on the ration of students to modern multimedia computers in the classroom. 4:1 is not nearly enough, 2:1 works for a hybrid classroom where the teacher can group the half the students at a time, and 1:1 is needed for highest effectiveness.
   

1. The academic testing was administered near the beginning and end of the academic year. The control groups were allowed to use technology products that may have been in their classrooms.
   

1. The results of the three to five different products used at the four different subject-grade groups (math 6th and Algebra; reading (1^st and 4^th) were averaged together in the group. We would expect that some products would increase academic performance; others might be less than effectual in using this method of implementation. Commingling of results would decrease the group’s maximum benefit.
   

1. The vendors provided technical assistance and training on using the products. There was no mention of education for the teachers on pedagogy and integration of digital curriculum into legacy education. The teachers liked the training in the summer and fall but had a lower degree of confidence after they began using the products in the classroom. Teachers could stop using the products or use them in ways not advised by the vendors. It takes up to four years of professional development and practice to transform a legacy teacher to full mastery as an elearning savvy teacher. The cost (assuming no support from a mentor-master teacher) is about $1500 a year. The use of first year eLearning teachers in training for the study would be expected to produce less than optimum results.
   

1. Lectures were less, and one-on-one teacher facilitation was up.
   

1. During the school year products were used on average 30 hours for 1^st grade reading (11% of the time) and 14 hours for 4^th grade reading (less than 10%). Sixth grade math was 17 hours per year (11% time) and Algebra was 15 hours (10% time). There was no data I could find that addresses if and what software was used in computer labs.
   

Until these issues and questions are addressed, I suggest that we continue on our advocacy trail. The one take-away is that with out a full elearning systems design and implementation, our goal of every student increasing academic performance a letter grade will continue to challenge us.