MEASURING UP AND STUDENT LEARNING
By Margaret A. Miller
|
Advisory Committee Members The Pew Project on College-Level Learning |
|
DAVID W. BRENEMAN University Professor and Dean Curry School of Education University of Virginia EMERSON J. ELLIOTT Director Program Standards and Evaluation National Council for Accreditation of Teacher Education PETER T. EWELL Vice President National Center for Higher Education Management Systems MARGARET A. MILLER Professor University of Virginia President Emerita American Association for Higher Education MICHELE SEIDL Officer Education Program The Pew Charitable Trusts VIRGINIA B. SMITH President Emerita Vassar College |
|
Participants The National Forum on College-Level Learning November 27-28, 2001 |
|
THE HONORABLE GARREY CARRUTHERS President and Chief Executive Officer Cimarron HMO, Inc. New Mexico GORDON K. DAVIES President Kentucky Council on Postsecondary Education THOMAS EHRLICH Senior Scholar Carnegie Foundation for the Advancement of Teaching California ROGER A. ENRICO Former Chairman and Chief Executive Officer PepsiCo., Inc. Texas THE HONORABLE JIM GERINGER Governor of Wyoming MILTON GOLDBERG Executive Vice President National Alliance of Business Washington, D.C. THE HONORABLE JAMES B. HUNT JR. Womble Carlyle Sandridge & Rice North Carolina GLENN R. JONES President and Chief Executive Officer Jones International, Ltd. Colorado ANN KIRSCHNER President and Chief Executive Officer Fathom Knowledge Network New York THE HONORABLE JOHN R. McKERNAN, JR. Vice Chairman Education Management Corporation Maine CHARLES MILLER Chairman Meridian National, Inc. Texas LILLIAN MONTOYA-RAEL Executive Director Regional Development Corporation New Mexico MICHAEL NETTLES Professor of Education and Public Policy University of Michigan STEFFEN E. PALKO Vice Chair and President XTO Energy, Inc. Texas THE HONORABLE PAUL E. PATTON Governor of Kentucky CHARLES B. REED Chancellor California State University SEAN C. RUSH General Manager Global Education Industries IBM Corporation Massachusetts EDWARD B. RUST, JR. Chairman, President, and Chief Executive Officer State Farm Mutual Illinois TED SANDERS President Education Commission of the States Colorado THE HONORABLE JACK SCOTT California State Senator KALA M. STROUP Commissioner of Higher Education Coordinating Board for Higher Education Missouri |
|
AS THE NATIONALLY TELEVISED SYMPOSIUM that announced the release of Measuring Up 2000 in November 2000 drew to a close, the participants
focused on topics that particularly captured their attention. Highest on their list was the Incomplete that the national report card had given to all states for student
learning. The leaders present-from business, education, and public policy-were astonished at and disturbed by how little Measuring Up could report about the
skills and knowledge of college students. |
 Margaret A. Miller is Project Director of the National Forum on College-Level Learning. |
- the American Association for Higher Education's longstanding Assessment Forum, which disseminates good practices in assessment by, among other means, an annual national conference;
- The Pew Charitable Trusts' Quality of Undergraduate Education project and various writing assessment projects, all linking assessment to the improvement of undergraduate education;
- Indiana University's National Survey of Student Engagement, a measure of good educational practice that has surveyed over 160,000 college students at over 470 colleges and universities (and a newly developed version of the survey for community colleges);
- the Collegiate Results Inventory, a survey developed by the Institute for Research on Higher Education at the University of Pennsylvania of almost 3,900 college graduates from 87 institutions;
- an effort by RAND and the Council on Aid to Education to develop a value-added assessment of undergraduate learning;
- the American Association of Colleges and Universities' general education assessment project; and
- regional accreditation associations' increasing insistence that institutional effectiveness be documented in terms of student learning.
A number of initiatives have also been undertaken from
outside the academy to determine how well adults are prepared for work, civic responsibilities, and family life. In 1990, for instance, then-Secretary of Labor Lynn Martin brought
together corporate, labor, and education leaders to form the Secretary's Commission on Achieving Necessary Skills (SCANS). Their work led to the report What Work Requires
of Schools. Another group, the National Skills Standards Board, continues to identify the knowledge, skills, and abilities students need to perform well in a growing range of
professions in the global economy. And the Equipped for the Future (EFF) project of the National Institute for Literacy has been attempting to determine the literacy needs of
adult Americans since 1993.The EFF project was prompted by the National Education
Goals, formulated in 1990 by then-President George Bush and a group of governors (including then-Governor Bill Clinton) and ratified by Congress in 1994. These goals
reflected a broad national consensus about what Americans should know and be able to do in the new millennium. Goal Six, the only one that focused on adults, said that
"by the year 2000, every adult American will be literate and will possess the knowledge and skills necessary to compete in a global economy and exercise the rights and
responsibilities of citizenship." In order to reach this goal, the President and governors set the following objective: "The proportion of college graduates who demonstrate
an advanced ability to think critically, communicate effectively, and solve problems will increase substantially."For several years after the National Goals were developed,
extensive discussions took place about how we might define and measure critical thinking, communication, and problem solving. But while a system of testing designed to
assess the literacy of schoolchildren was developed, no nationwide attempt was made to do so for college students.Instead, the most concerted attempts to assess the
learning of college students were occurring within individual states, where the primary responsibility for education lies. As states increased their investments in higher
education (from $21 billion in 1980 to $64 billion in 2001), their interest in those investments grew as well. From the late 1980s through the 1990s, the states established a
variety of assessment programs in their public colleges and universities-some focused on individual student certification, others on institutional improvement, and still others
on accountability. Consequently, several states now have information about the learning of students in their public colleges and universities, and some of these even have
comparable information across their public institutions.But few states, if any, know about the learning of their
graduates of private colleges-or about the intellectual capabilities of their college-educated residents, regardless of where they were educated. Moreover, the information
that states do gather about collegiate learning is specific to each state; it cannot be used to compare performance relative to other states. As Measuring Up 2000
made clear, it is only in the context of these kinds of comparisons that a state can know whether its level of performance is good or bad news.After the publication of Measuring Up 2000,
The Pew Charitable Trusts sponsored a project to investigate how to address the issue of college-level learning. As the project director, and counseled by an advisory
committee (see sidebar), I interviewed higher education leaders across the country. I asked each of them whether or not this was the time to undertake a systematic,
nationwide assessment of college-level learning, and if so, how we might go about it. It soon became clear that selecting the questions to be addressed by such an
assessment would determine the worth of the undertaking, the kinds of information to be sought, the appropriate groups to assess, and the policy uses to which the
information might be put.|
For instance: Do we care most about certifying the
performance of individual students? Are we interested in how well, individually or collectively, institutions in the states foster learning? Or do we want to gauge the intellectual
skills of the college-educated residents in each state-the educational capital they bring to bear on the state's economic and civic problems-wherever they might have been
educated? If the interest is in individual certification, every student should be tested on skills relevant to that certification. If the focus is on the effectiveness of a state's
institutions, a representative sample of students in each institution or state should be assessed regarding what they learned in college. To get at the educational capital
question-that is, what productive value do college-educated residents add to a state's resource base?-measures of the functional intellectual skills of the population at large
are needed. And the policy entailments of each of these questions also differ: individual certification and institutional effectiveness have most relevance for higher education
policy, whereas the educational capital question might be addressed through policies on adult literacy or even economic development.
|
|
All of these discussions about assessing college-level
learning culminated in November 2001, when a group of business, higher education, and policy leaders met at the National Forum on College-Level Learning in Purchase, New
York (see sidebar). The group considered the same questions that had been raised in the preceding months: Was it time to assess college-level learning in such a way as to
permit state-by-state comparisons? If so, what questions should such an initiative answer?Despite the considerable financial challenges that the
states currently face, the forum participants concluded that such an effort was so long overdue that it should begin now. They also agreed that the most pressing questions
that might be addressed were those about the collective effectiveness of each state's institutions and the educational capital embodied in its college-educated residents.
Finally, they made some suggestions about how the initiative might proceed, both in the short and long-term.The most immediate strategy they endorsed was to
collect information from existing licensure tests and graduate school and professional school admissions examinations. While recognizing the limitations of such an approach
(the unrepresentative nature of the test-taker groups being the most important problem), the participants concluded that, given the credibility of these tests with both
those who take and those who rely on them, they were a good place to start. They also thought that other instruments that have already been widely administered, such as
the National Assessment of Adult Literacy (NAAL) and the National Survey of Student Engagement (NSSE), could provide additional information.Other measures also exist that have yet to be
administered systematically across the states. The forum participants suggested that a group of states pilot a model using some of these measures, such as the
Collegiate Results Survey and WorkKeys, a series of tests of the intellectual capabilities of people moving into the workplace. The forum thought that in the long run,
business and higher education should work together to develop a new instrument that would measure functional intellectual skills of college-educated people, such as those
identified in National Education Goal Six: the capacities to communicate, solve problems, and think critically.Since the forum, the advisory committee to the project,
working with the National Center for Public Policy and Higher Education, has been developing a prototype for measuring college-level learning and testing it using data from
Kentucky. (For a full description of the prototype, see "Constructing Indicators: A Proposal for Discussion") The model, limited to existing sources of information, is based on results
from licensure and admissions tests, information generated by the Kentucky administration of the National Adult Literacy Survey, and results previously generated by the National
Survey of Student Engagement. All of this information is placed in the context of national results on those measures.|
Meanwhile, a new two-year grant from The Pew
Charitable Trusts will extend the project to several additional states. The goal in this phase will be to improve the quality of the data and add more measures. This pilot
should provide a better understanding of how to assess the educational capital that states have in their college-educated residents and the effectiveness of their higher
education systems in contributing to that capital. The results will be described in Measuring Up 2004. If the pilot has been successful, and if the National Center
is able to gather the same information from most other states by 2006, it should be able to assign grades on college-level learning in Measuring Up 2006.Although the current grant does not support it, work
should continue during and beyond the grant period to develop a new state-level measure of the general intellectual skills of the college-educated. The suggestion by forum
participants that this be a collaborative effort of higher education and business is a good one. Just as war is too important to be left to the generals, so education is too
important to be left to the educators. We need the means to measure our states' and nation's collective capacity to meet the challenges of the coming era. In the National
Assessment of Educational Progress we have charted the uneven, sometimes faltering, and largely unchanging student learning in primary and secondary education. We
need to track learning beyond high school as well, and it is not too soon to begin.
|
|