Grading Student Learning: You Have to Start Somewhere
By Peter T. Ewell

Peter T. Ewell is Vice President of the National Center for Higher Education Management Systems.

IN MEASURING UP 2000, the National Center for Public Policy and Higher Education awarded all states an Incomplete in student learning because there were no common benchmarks that would have allowed for meaningful state-to-state comparisons. This year, the National Center made the same decision, for the same reason. Two years was not enough time for states and national organizations to create the assessment tools and systems needed to collect credible information on college-level learning.

Two important developments, however, are causes for optimism. First, the report card issued in 2000 has, as we had hoped, stimulated valuable discussions among governors, state legislators, and business leaders about college-level learning. Some governors and legislators were startled to realize that their states do not have answers to key questions about the knowledge and skills of their residents-information that is important for developing an adaptive workforce, maintaining meaningful citizen participation, and promoting active community life. Many business leaders were startled as well, finding it "outrageous" that a report card on higher education could say nothing about the knowledge and skills of those who had completed at least some education beyond high school. Second, these discussions among policymakers have provoked new thinking about approaches to constructing statewide indicators of adults' competencies in such areas as critical thinking, communications skills, quantitative literacy, and problem solving.

EDUCATIONAL CAPITAL: A KEY STATE RESOURCE

In November 2001, The Pew Charitable Trusts sponsored a National Forum on College-Level Learning, which brought together a high-profile group of business, political, and educational leaders to discuss the value of creating the infrastructure for assessing college-level learning. During the forum, a consensus developed that this was a worthy goal-but not just for the sake of knowing something about academic achievement. Instead, policymakers in both the public and private sectors emphasized the importance of being able to monitor "educational capital"-the store of our nation's high-level knowledge and skills, which affects every contour of our political, economic, social, and cultural life.

In our personal life, making wise choices that affect our well-being has become more and more challenging. For instance, people must make complex choices among health care plans, telephone services, credit card companies, and the like. Possessing the intellectual tools of reasoned judgment and information literacy is no longer a luxury. Being deprived of them is a notable social injustice. In public life, meanwhile, the challenge of citizenship is increasingly formidable. To make decisions about national issues, citizens more than ever need to become more global in their thinking, to understand the perspectives and limits of religion and ideology, and to think historically. Meanwhile, the 21st century workplace demands sophisticated problem-solving skills, while the dramatic demise of Enron reminds us that inattention to the ethical dimension of work can have profound social and economic consequences.

States need to be able to assess the educational level of their residents, to help guide new investment in higher education, to identify subpopulations that require special attention, and to ensure equitable access to educational opportunities. Data on educational capital could also be used to build support for new initiatives in higher education, much in the way that national leaders have used international rankings in science and mathematics to rally support for initiatives to address deficiencies in those disciplines.

Taking a statewide approach to assessing educational capital would sidestep an obstacle that has derailed earlier attempts to measure college-level learning: Colleges and universities resist attempts that seek to compare institutions, because they fear the repercussions for institutions that perform poorly. But if the objective is to measure the educational capital of the state-for the purpose of creating state policies that will improve the performance of graduates from all institutions-the dynamic is different. Institutions could participate positively in an effort to define high national standards and help all states set their priorities for higher education.

BASIC QUESTIONS ABOUT EDUCATIONAL CAPITAL

In order to craft their state's higher education policies, state leaders need information that will help them answer the following questions.

. . . . "Many business leaders were startled . . . that a report card on higher education could say nothing about the knowledge and skills of those who had completed at least some education beyond high school." . . . .

What are the current ability levels of state residents, especially those who have attended college? State policymakers need information about their state's general store of educational capital and also about strengths and weaknesses in particular kinds of abilities-for example, communications skills or quantitative skills-and the distribution of these abilities across geographic regions and among demographic groups. This information would help policymakers identify current deficiencies and project the increment in educational capital that their state could realize by making investments that increase college enrollments.

What contributions do the state's colleges and universities make to the stock of educational capital? The colleges and universities, both public and private, located within a state are the principal partners in producing and increasing educational capital. State leaders therefore need to know about the competencies of students who graduate from these institutions. Are they well-enough prepared to obtain appropriate employment? to enroll in graduate studies? to enter professional schools? State leaders also want to know whether their state is a net importer or net exporter of educational capital: Is their state retaining a good share of its new college graduates or are large numbers of them moving out? Is their state attracting and retaining college graduates from other states?

How do the state's learning outcomes compare to national standards and to the outcomes achieved in other states? Many states set their own standards for educational achievement. For example, every state has its own standards for licensing schoolteachers; even when several states use the same test, each sets its own passing score. To determine whether their state's college graduates meet the highest standards, policymakers need to know not only how many residents take postgraduate admissions examinations or professional licensing examinations, but also how well these residents score in comparison to national standards and to the scores of residents in the best-performing states.

What efforts are being made by the state's public and private colleges to promote better learning? Decades of research have identified "good practices" in undergraduate instruction that significantly enhance learning; these include student-faculty interaction, high levels of academic challenge, active and collaborative learning, and a supportive campus environment. Policymakers can provide institutions with support and incentives for adopting practices that enhance student achievement.

Comparative data in each of these domains can help states determine where they stand and in which areas investment of additional resources is likely to make the largest difference. Comparative data on educational capital can also help state leaders build a public case for making these investments.

MEASURING EDUCATIONAL CAPITAL

The National Center's decision to not assign grades for student learning reflects the insufficiency and unevenness of the available data. Creating instruments that can be used to assess such skills as critical thinking and problem solving will take considerable effort, time, and money. But there are also stores of useful data that could be incorporated into the analysis of educational capital if ready access were provided to researchers. And some meaningful comparisons could be made if more states chose to participate in important data-collection initiatives that are already under way.

The following review suggests how indicators of state educational capital might be constructed and what information these indicators could yield.

Population Assessments

The only national measure that examines the abilities of the general population is the National Adult Literacy Survey (NALS), last administered in 1992. This measure is due to be updated in 2003; the new instrument will be called the National Assessment of Adult Literacy (NAAL). These assessments address three "literacies" that are critical for effective functioning in the workplace and society: prose, document, and quantitative. Both instruments allow comparisons between the performance of adults who attended college and those who did not, although the levels of ability examined by NALS and NAAL are modest. (Results of the 1992 survey suggest that not all college graduates perform as well as we would like them to, even on these basic tasks.) Oversampling to obtain valid and reliable state-level estimates is technically possible, and many states have done so. State-to-state comparisons based on NALS/NAAL are limited, however, because some states choose not to participate.

Data on educational attainment (years of schooling) are available from the decennial census. Educational attainment is, of course, only a very broad and indirect measure of educational capital.

	What we can do now is use the NALS information on literacies to examine the relative abilities of those who attended college and those who did not-both absolutely and in terms of the "added value" of college attendance in a particular state.
	What is needed for the future is an instrument for examining levels of ability that are more sophisticated than those measured by NAAL, and a means to ensure that as many states as possible participate.

Assessments of the Abilities of College Graduates

There are no measures of college-level learning that are systematically administered to the nation's college graduates. A few states collect information of this kind, but only for public institutions. The examinations used in these states are also rather limited in assessing higher-order abilities like critical thinking and problem solving. Ideally, we would want assessments that measure both "academic" applications of advanced skills (in physics or history, for instance) as well as "real world" applications (for example, those useful in workplace and other settings).

The model for this instrument might be the widely respected National Assessment of Educational Progress (NAEP), currently used to measure learning in K-12 schools. Possible candidates include the Tasks for Critical Thinking examination, developed by New Jersey in the late 1980s; the Work Keys examination program developed by ACT to assess work-related skills (though it would be desirable to adapt this exam to test higher levels of ability); and the "Value-Added" assessment battery now being pilot-tested by the Council on Aid to Education (CAE), a subsidiary of the RAND Corporation. All three of these approaches go beyond simple multiple-choice testing.

	What we can do now is administer some of these assessments on statewide samples of students to explore their appropriateness.
	What is needed for the future is a serious development effort to create a counterpart to the NAEP for higher education within the next ten years.

Licensure and Admissions Examinations

Many college graduates take various examinations as a prerequisite for entrance to graduate study or a profession. Scores on these examinations, however, cannot be used to directly measure a state's educational capital because each test is designed to meet a specific set of purposes. Moreover, different proportions of college graduates in different states take these examinations, and these differences in participation rates will affect any composite measure of statewide performance. These measurement difficulties can be mitigated somewhat by looking at the number of individuals in each state who achieve a particular level of attainment rather than calculating each state's mean performance scores.

There are also practical obstacles to using scores from these examinations, because each is governed by a different authority with its own policies about access to data. Some authorities release results only in aggregate form, using statistical measures unique to that examination. Others allow researchers to access the raw data files, which permits the direct calculation of comparable statistics, and some will supply researchers with the statistics they request. A few do not allow access to their data in any form.

	What we can do now is use information from available licensure and admissions examinations to demonstrate the concept of "readiness for advanced practice" as an element of educational capital in selected areas.
	What is needed for the future is to extend access to licensure and admissions examination information so that it can be used for public policy purposes.

Indirect Measures Several surveys containing questions that indirectly measure student learning are now administered to college students across the nation. The National Survey of Student Engagement (NSSE) and the Community College Survey of Student Engagement (CCSSE) contain items that tap the "good practices" known to promote better collegiate-level learning. The NSSE has been administered to large numbers of about-to-graduate four-year college seniors across many states; the CCSSE is just getting started. The Collegiate Results Survey (CRS), administered to college graduates by Peterson's Guide, includes items on self-reported learning and current activities. The results of these surveys, although they represent only indirect measures of college-level learning, can be used to supplement direct measures.   What we can do now is compile state-level results from the NSSE on educational "good practices" for those states that have sufficient data coverage.   What is needed for the future is to extend the administration of the NSSE and CCSSE to all states, and administer the CRS (or similar surveys) to national samples of college graduates.

. . . . "States need to be able to assess the educational level of their residents, to help guide new investment in higher education, to identify subpopulations that require special attention, and to ensure equitable access to educational opportunities." . . . .

LOOKING FORWARD

Creating new instruments to reliably assess college-level learning will require considerable time and effort; part of the problem is that we still have inadequate instruments for assessing skills like critical thinking and problem solving. But individual states can improve their ability to monitor their educational capital by taking two steps right now: (1) states that have not participated in existing national surveys such as the NAAL could elect to do so, and (2) states could work proactively to induce testing and licensing authorities to open their databases to researchers seeking to improve the state's store of policy-relevant information. With more data in hand, state leaders could begin to realistically assess the mix of key abilities among their citizens in relation to state economic and workforce development plans. They could use these data to help persuade firms in key industries to locate in their states, direct state investments to remedy identified gaps in workforce skills, and adjust their plans to respond to changing economic conditions on an ongoing basis. These immediate actions will admittedly not give us the measures we will ultimately need to monitor and improve the nation's store of educational capital. But we have to start somewhere.