Commentary: Current Year

Grading Learning: Progress and Prospects

By Peter T. Ewell

It has been six years since Measuring Up 2000 awarded all 50 states an “Incomplete” in Learning because comparable data were not available to make meaningful state-to-state comparisons in this category. Since then, much has been accomplished, but a great deal more remains to be done.

Measuring Up 2004 for the first time awarded a “Plus” in Learning to five states—Illinois, Kentucky, Nevada, Oklahoma, and South Carolina—because of their pioneering participation in a national demonstration project conducted by the National Forum on College-Level Learning and funded by the Pew Charitable Trusts. 1 These states are joined in 2006 by four more—Maryland, Massachusetts, Missouri, and New York—that participated in the State Assessment of Adult Literacy (SAAL), a state-level version of the National Assessment of Adult Literacy (NAAL) conducted in 2003. 2

Measuring Up 2006 also features a 50-state demonstration of one component of Measuring Up’s Learning model—“Graduates Ready for Advanced Practice.” This component assesses the quality of each state’s “educational capital” by comparing the information provided by the hundreds of thousands of professional licensure and graduate admissions examinations completed by the nation’s college graduates each year (see accompanying text box for a description of the Measuring Up Learning model).

This slow but steady evolution has been accompanied by growing national interest in taking stock of college-level learning. In the broadest terms, events of the opening decade of the new millennium highlight the nation’s competitive challenge in producing and harnessing our educational capital. Where we once led the world, the United States is now only seventh in the proportion of young adults (ages 25 to 34) who have earned a postsecondary degree—and we have been overtaken in just one decade. Meanwhile, results of the NAAL, although much debated, suggest that the abilities of the nation’s college graduates have slipped during the same period. 3 Spurred by these findings, bodies like the Secretary of Education’s Commission on the Future of Higher Education have underscored the need, among many others, for more systematic information about what college graduates know and can do.

The Learning Model

Note: Measures included under the first two clusters are available nationally and can be calculated for all 50 states. Measures included in the third will require special datacollection efforts similar to those undertaken by the five demonstration project states in 2004.

Both states and institutions have responded positively to these challenges, though progress has been uneven. Kentucky and Oklahoma have already committed to repeating the Measuring Up Learning model next year. States like South Dakota and Tennessee—as well as some large public systems like the City University of New York (CUNY)—continue to examine learning directly. While such examination results cannot be benchmarked to other states and may be used to ground questionable comparisons among institutions with quite different missions, the basic intent of such efforts is admirable. Meanwhile, states like Virginia and Utah—among many others—continue to encourage institutions to assess learning locally against commonly stated commonly-stated outcomes in key categories such as communication and critical thinking. Institutional efforts to assess learning and publicly report results are being further reinforced by regional accrediting organizations as well as by the public commitments to assess learning made by institutional membership organizations such as the National Association of State Universities and Land-Grant Colleges (NASULGC) and the American Association of State Colleges and Universities (AASCU). These are clear signs of progress and they should be justly recognized.

But there are major challenges ahead. When the National Forum convened in 2001 to consider how Measuring Up might incorporate Learning assessments, it concluded that the best immediate course was to exploit existing measures. The Learning model used in Measuring Up 2004 and extended in this edition is consistent with this advice. But this by no means implies that these measures are the best we can ultimately get. Indeed, the conceptual categories that define the model can accommodate far better data. For example, the 2003 NAAL provides the basis for an updated national benchmark in “Literacy Levels of the State’s Residents.” But future administrations of this important assessment ought to be based on sample sizes that allow credible state-level estimates, and more states should participate in SAAL. In addition, it is especially unfortunate that we have had to wait for more than a decade for the update on the nation’s literacy that NAAL provides.

Licensure and graduate admission examination results—now available in the aggregate for all 50 states—enable full calculation of “Graduates Ready for Advanced Practice.” But not all graduates take such examinations, and indirect estimates such as these will always be affected by unknown factors determining who takes such tests in each state.

Results for “Performance of College Graduates” depend on the availability of suitable assessments and sufficient investments in state-level data collection to ensure that results are robust and reliable. The Collegiate Learning Assessment (CLA) administered to four-year students and ACT, Inc.’s WorkKeys assessments administered to two-year students in 2004 were among the best of their kind then available. But assessment technology will continue to develop and should be exploited. More authentic and comprehensive assessments—ideally constructed to examine how much students have grown during the college experience—are badly needed. Equally necessary are incentives for states and institutions to participate in such efforts and use their results honestly.

The principal goal of the National Forum’s five-state demonstration project on Learning, reported in Measuring Up 2004 and extended here, was to prove the feasibility of this approach. We now face the far more serious long-term task of filling out the model with constantly improving measures, administered to increasingly larger samples of students and citizens, and extended to all 50 states.

Peter T. Ewell
Vice President of the National Center for Higher Education Management Systems and
member of the National Advisory Group for Measuring Up 2006.

The Learning Model

The Learning category is constructed as the other five performance categories of Measuring Up have been, with indicators that are grouped in three overall themes, each of which is weighted (see parentheses) and reflects a particular dimension of state performance:

1. Literacy Levels of the State’s Residents (25%). This cluster of indicators examines the proportion of college-educated citizens who achieve high levels of literacy. It directly addresses the question, “What are the abilities of the state’s college-educated population?” originally posed in Measuring Up 2000.
   
   F or Measuring Up 2006, the data are drawn from the State Assessment of Adult Literacy (SAAL) administered to adults with an associate’s or a bachelor’s degree. The SAAL was administered in five states in 2003—Kentucky, Maryland, Massachusetts, Missouri, and New York—in parallel with the National Assessment of Adult Literacy (NAAL). The SAAL poses real-world tests or problems that require respondents to read and interpret texts (prose), to obtain or act on information contained in tabular and graphic displays (document), and to understand numbers and graphs, and perform calculations (quantitative).
   

2. Graduates Ready for Advanced Practice (25%). The indicators in this theme reflect the contributions higher education makes to a state’s stock of “educational capital” by examining the proportion of the state’s two-year and four-year college graduates who are ready for advance practice in the form of professional licensure or graduate study. It addresses Measuring Up 2000’s policy question, “To what extent do colleges and universities in the state educate students to contribute to the workforce?”
   
   For Measuring Up 2006, the measures are based on the number of college graduates within each state who have demonstrated their readiness for advanced practice by a) taking and passing a national examination required to enter a licensed profession such as nursing or physical therapy, b) taking a nationally recognized graduate admissions exam like the Graduate Record Examination (GRE) 4 or the Medical College Admissions Test (MCAT) and earning a nationally competitive score, or c) taking and passing a teacher licensure examination in the state in which they graduated college. Each of these measures is presented as a proportion of the total number of bachelor’s and associate’s degrees granted in the state during the same time period.
   

3. Performance of College Graduates (50%). This cluster of indicators focuses on the quality of the state’s higher education “product” by addressing the all-important question, “How effectively can graduates of two- and four-year colleges and universities in the state communicate and solve problems?”
   
   Measuring Up 2006 presents the same results that were presented in Measuring Up 2004 for the five states that participated in the National Forum on College-Level Learning’s demonstration project: Illinois, Kentucky, Nevada, Oklahoma, and South Carolina. The measures used consisted of two sets of assessments, the Collegiate Learning Assessment (CLA) for four-year institutions, and the ACT WorkKeys assessment for two-year colleges. The CLA is an innovative exam that goes beyond multiple-choice testing by posing real-world tasks that a student is asked to understand and solve. For example, students could be asked to draw scientific conclusions from a body of evidence in biology or examine historical conclusions based on original documents. They might be asked to prepare a persuasive essay, in which they analyze and refute a written argument by means of logic and evidence. The ACT WorkKeys assessment examines what students can do with what they know. Items that assess reading comprehension and skills in locating information, for instance, might require students to extract information from a document or a set of instructions; questions in applied mathematics might test their ability to use mathematical concepts and skills such as probability or estimation in real-world settings. The WorkKeys writing assessment requires students to prepare an original essay in a business situation.

In order to evaluate state performance, the values for each indicator within these three themes are compared to a common standard. The standard used for Measuring Up 2006 is the national average on each measure.

1. For a full report, see Measuring Up on College-Level Learning here.

2 Kentucky, a National Forum demonstration state, also participated in the 2003 SAAL.

3 Justin D. Baer, Andrea L. Cook, and Stéphane Baldi, The Literacy of America’s College Students (Washington, D.C.: The National Survey of America’s College Students, American Institutes for Research, January 2006).

4 The National Center is grateful to Kurt Landgraf, President of the Educational Testing Service (ETS) for making state-level GRE scores available for the first time.