An essay written for my Preliminary examination at Texas A&M University, Fall 2009.
Literature Review & Discussion
How can researchers try to assess the quality of curriculum materials? I would like to make the initial assumption that, by en large, “curriculum materials” correspond to textbooks. That is not completely true because in addition to textbooks, mathematics teachers often employ workbooks, manipulatives, and charts . Furthermore, modern schools will often complement classroom instruction with computer based learning.
The curriculum itself is a composite entity. According to Senk and Thompson (2003, note 2, pp. 26-27)
The mathematical content specified in a set of recommendations or in a set of instructional materials can be viewed as the intended curriculum. The mathematical content actually studied in an individual classroom can be viewed as the enacted curriculum. The mathematical content actually learned by the students can be viewed as the achieved curriculum. (note 2, p. 27)
There is another type of curriculum, the “assessed curriculum” . The author considers the intended curriculum the “content specified by the standards,” the enacted curriculum the “content taught,” and the assessed curriculum the “content tested.” However, the assessed curriculum should not be confused with “curriculum assessment,” defined as the measurement of the “academic content of the intended, enacted, assessed curricula as well as the content similarities and differences among them” (p. 141).
In line with these classifications then, the assessment of the quality of curriculum materials would correspond to the assessment of the intended curriculum. This assessment could be employed to determine the quality of textbooks if we equate the intended curriculum with the textbooks themselves as Senk and Thompson (2003) have done.
In principle it is perfectly valid to evaluate a textbook solely within the context of the personal parameters of the researcher. The mathematician Richard Askey:2001aa examined a series of middle school mathematics textbooks, called Connected Mathematics , that had received the highest rating by Project 2061 ( http://www.project2061.org/publications/textbook/mgmth/report/part2.htm#Textbooks). The author examined the content of these textbooks and was dismayed at finding many conceptual errors in both the student and the teacher editions. Project 2061 is an AAAS American Association for the Advancement of Science founded project created in 1985 “to help all Americans become literate in science, mathematics, and technology.” (http://www.project2061.org/about/default.htm)
This evaluation performed by Project 2061 is an example of a direct curriculum assessment based on the intended curriculum. The general procedure for this kind of assessment is to establish a correspondence between the standards and the textbook. One can implement this general concept in several different forms. Porter (2006, pp. 141–144) presented the following procedure. The researchers start by dissociating the standards document(s) into basic components and arranging them according to some sort of classification. The textbook under examination will be similarly decomposed into basic blocks that are organized using the same classification scheme. Then, the researchers attempt make a one-to-one mapping of the components. By calculating the percentages of correspondence relative to the complete book as well as its topics it is possible to determine how well the textbook matches the standards.
Project 2061 evaluated several middle school mathematics textbooks in light of an intended curriculum of reference, in this case the NCTM Curriculum and Evaluation Standards for School Mathematics and the AAAS Benchmarks for Science Literacy . The researchers describe their procedure as a series of steps. At Step 1 “the analysts examine each textbook activity – a lesson or part of a lesson – that matches the content of the benchmark.” At Step 2 the “Analysts determine the extent to which an activity addresses the benchmark concept or skill.” At Step 3 the “Analysts decide which of the 24 instructional criteria apply to the activities the have identified throughout the textbook.” These criteria would correspond to the above mentioned classification of standard components. Finally at Step 4 “Based on the indicators met, analysts rate the activity on a scale of 0 to 3 for each criterion.” At the end a composite score is calculated.
Another method by which a textbook can be evaluated is an indirect method, which is much more laborious. If it can be assumed that the intended, enacted, and achieved curricula are closely related, then we can gauge a textbook by the assessed curriculum. Such type of evaluation was performed by Schoen (2003, pp. 314-340) for the Core-Plus Mathematics Project. The project had developed a comprehensive high school mathematics curriculum (http://www.wmich.edu/cpmp/evaluation.html). To facilitate this type of analysis the curriculum developers built into the material itself its own assessment (p. 315). The authors analyzed the data and concluded that there is “strong evidence in support of the feasibility of the curriculum and of the Standards-oriented reform generally.” (p. 341). However, both of their graphs lacked error bars, not all their tables showed t values, and no effect sizes were given.
The National Research Council (2004) sponsored a meta-analysis of the research literature on about 19 K-12 mathematics curricula. They classified 192 studies into type of study, content analysis, case studies, comparative studies, and syntheses. The conclusion of this analysis was that it is not possible, with reference to these 19 curricula to “determine the effectiveness of individual programs with a high degree of certainty, due to the restricted number of studies for any particular curriculum, limitations in the array of methods used, and the uneven quality of the studies” (p. 3).
Summary Critical Analysis
Any assessment is open to criticism and so is the assessment of textbooks. Even the direct method, where textbooks are compared to the standards, can be contested appealing to the threads of validity at each step of the procedure.
Any research project that entails the conversion of textual data into quantitative form has issues that go beyond those of randomization and sampling. There are problems at each step of the way starting with the decomposition of the concepts into components, and then their classification. How can we make this process consistent and replicable? How do we measure “closeness” to a curriculum objective of a lesson?
All these issues are more pervasive in the case of an indirect method of assessment, when the textbook is evaluated in light of students’ test results because there are more steps involved. We have to work all the way back from test results to classroom instruction, to textbooks, to standards.
Previously, mathematicians who wrote textbooks worked almost completely independently from each other and there were no official standards. The standard was the consensus of the mathematicians of that time who used a small set of textbooks. It a certain sense at that time instruction was much more standardized than it is today. All universities regardless of where they were located used the same textbooks in the same language (Latin). A mathematics teacher could with no difficulty at all move from a university in Spain to one in Poland, for example. He would not have to worry about choosing a different textbook or change his lectures in any way. When the mathematics teacher and mathematician Nicolo Tartaglia wanted to publish a geometry textbook in his native language, Italian, he did not write one from scratch, but translated the at that time one and only geometry textbook, The Elements of Euclid . His original material was mostly limited to some additions to the original text written at the end of each theorem or as introductions to chapters. Later, when new mathematics, such as the `new algebra’ was developed in Europe, new and original textbooks had to be written. François Viéte (1591) published an algebra textbook entitled In Artem Analyticem Isagoge that became very popular in Europe.
Today, while mathematics textbooks in universities are still written in this fashion, no K-12 mathematics textbook is written anymore by a single person. In addition, the writers are not free to follow their ideas regarding what a textbook should contain, but have to follow official standards. In the USA these standards consist of a complex assembly of professional (NCTM), state and federal guidelines. Further constraints are the financial necessities of the publishers and the choices made by state and school district textbook selection committees.
The Internet has allowed educators to collaborate among themselves as has never been possible before. One of the new developments is the production of open textbooks. The Internet allows a group of writers to produce comprehensive textbooks without ever meeting face-to-face. In addition, through the Internet these textbooks can easily be distributed by downloading the text in PDF or Ebook format. Finally, these text can be read by the user on laptops or electronic readers, and even printed and bound. One of these initiatives is the project called “Free High School Science Texts” (http://www.fhsst.org). Among the downloads are grade 10, 11, and 12 mathematics textbooks.
Should the K-12 mathema.tics teachers in the USA support these initiatives? Could NCTM take the lead in fostering the production of free online textbooks? Besides having the effect of making mathematics education more affordable for school districts, another advantage from a curriculum point of view would be the short-circuiting of the slow, laborious and expensive standards to textbook process. How would that effect the assessment of these free collaborative electronic textbooks? This mode of production of textbooks would decouple mathematics teachers from the publishing companies. Then, how should NCTM have to position itself in relationship with the publishing companies? Would there be conflicts of interest?
These collaborative mathematics textbook are developed in the open, for anyone to observe. Would this if not avoid, at least abate the Mathematics Wars? Parent organizations for instance could monitor in real time the writing of these books and quickly submit their feedback. Up to now the time distance between the writing of a textbooks and the reaction of the public has been large. This has caused a large waste of resources. Countless work-hours of textbook composition are wasted when a series of textbooks is abandoned and replaced by a new one.
Consider the K-12 mathematics teaching during the 20th century. At the beginning it was traditional and somewhat elitist. The Sputnik brought about the “New Math” which was very rigorous and abstract. That was later abandoned for a “Return to Basics.” In 1983 “A nation at risk” was published and NCTM-inspired textbooks were written, such as the above mentioned Connected Mathematics textbooks. That produced another reaction that wants to bring us back to where we started, that is the traditional mathematics curriculum. This time, however, it is not politically correct to be elitist. Thus, “No child left behind” instructs that all students must have access to this type of rigorous mathematics and that curricula have to be developed using “science based research.”
The irony of this all is that while since the time of Euler and Gauss mathematics at K-12 levels has not changed that much, a wide variety of textbooks have been frantically adopted and then soon discarded. There is no inherent need to do so.
Political intervention in mathematics teaching is a sign that the public is concerned about the state of mathematics teaching, or more precisely about the curriculum. Would an open collaborative process for the writing and updating of textbooks allay this concern?
What good is it for mathematics research to busy itself with curriculum projects and their assessment when they are but implemented for a short time? According to National Research Council (2004, p. 1) the National Science Foundation spent 93 million to produce 13 different curriculum materials. Would it not be more productive to address the reasons that cause this high textbook turnover? That is the approach that I would take in curriculum assessment.
AAAS. (1993). Benchmarks for science literacy. Oxford, UK: Oxford Univesity Press.
Askey, R. (2001). Good intentions are not enough. In T. Loveless (Ed.), The great curriculum debate: How should we teach reading and math? (pp. 163–183). Washington, DC: Brookings Institution Press.
Lappan, G., Fey, J. T., Fitzgerald, W. M., Friel, S. N., & Phillips, E. D. (1998). Connected mathematics. Upper Saddle River, NJ: Dale Seymour Publications.
National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.
National Research Council. (2004). On evaluating curricular effectiveness: Judging the quality of K-12 mathematics evaluations (J. Confrey & V. Stohl, Eds.). Washington, DC: National Academy Press. (Available online at http://http://www.nap.edu/catalog.php?record id=11025#toc)
Porter, A. C. (2006). Curriculum assessment. In J. L. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 141–160). Mahwah, NJ: Lawrence Erlbaum Associates.
Schoen, H. L., & Hirsch, C. R. (2003). The Core-Plus Mathematics Project: Perspectives and student achievement. In S. L. Senk & D. R. Thompson (Eds.), Standards-based school mathematics curricula – What are they? What do students learn? (pp. 311–343). Mahwah, New Jersey: Lawrence Erlbaum Associates.
Senk, S. L., & Thompson, D. R. (2003). School mathematics curricula: Recommendations and issues. In S. L. Senk & D. R. Thompson (Eds.), Standards-based school mathematics curricula – What are they? What do students learn? (pp. 3–27). Mahwah, New Jersey: Lawrence Erlbaum Associates.
Tartaglia, N. (1565). Euclide megarese acutissimo philosospho, solo introduttore delle scientie mathematice. diligentemente rassettato, et all integrita ridotto, per il degno professore di tal scientie Nicolo Tartalea brisciano. Secondo le due tradottioni. Con vna ampla espositione dell istesso tradottore di nuouo aggiunta. Venice, Italy: Curtio Troiano.
Viéte, F. (1591). In artem analyticem isagoge. Tours, France: Meteyer.