Measuring Up 2002: The State-by-State Report Card for Higher Education

CONSTRUCTING INDICATORS: A PROPOSAL FOR DISCUSSION
CONSTRUCTING INDICATORS OF EDUCATIONAL CAPITAL
It is possible to illustrate a preliminary set of indicators of educational capital, despite incomplete and unsatisfactory data. The National Center selected Kentucky for this example because it has collected and made available more data than most states and, perhaps more importantly, because its state leaders believe that benchmarking state performance in collegiate learning is a valuable tool for policymaking.
The indicators of educational capital used in this illustration follow from the four policy questions discussed in Peter Ewell's essay. Each indicator is calculated for Kentucky and then compared to a national benchmark for all 50 states. The proposed weights represent informed speculation on the relative value of the indicators in terms of credibility and policy importance. These weights would need to be reviewed, of course, as new information becomes available. For example, if a college-level counterpart to the NAEP were to be developed and its results were deemed credible, the proposed weighting could be reduced for the more problematic data supplied by admissions and licensure tests.

Abilities of College-Educated Residents (Weight = 20%)

  • College-educated residents with advanced literacy. Calculated as the number of individuals who have completed at least some college-level work and who score four or higher (out of a possible five) on the NALS assessment, compared to the total number of individuals who have completed at least some college-level work. (Indicator Weight = 15%)

  • Value-added of college attendance. Calculated as the difference between the literacy levels on the NALS assessment of those individuals who have completed at least some college-level work and those who only completed high school. (Indicator Weight = 5%)
These indicators are calculated independently for each of the three subscores of the NALS-prose literacy, document literacy, and quantitative literacy-and the Kentucky Adult Literacy Survey (KALS), an equivalent assessment that uses the same methodology as the NALS.

Institutional Contributions to Educational Capital (Weight = 35%)

  • College graduates with advanced abilities. This indicator cannot be calculated for Kentucky or the nation in 2002 because there is no suitable assessment instrument. The measure would be calculated as the number of college graduates scoring above a particular level on an appropriate assessment, compared to the total number of college graduates. For example, results might be reported in terms of key abilities (such as communication and problem solving) identified by the National Education Goals Panel. (Indicator Weight = 20%)

  • College graduates ready for advanced practice. Calculated as the number of college graduates who have either passed a licensure examination or achieved a nationally competitive score on a standardized graduate admissions examination, compared to the total number of college graduates. Ideally, these indicators would be reported separately for (a) licensure examinations at the four-year college level, (b) licensure examinations at the two-year college level, and (c) graduate admissions tests. (Indicator Weight = 10%)

  • Graduates reporting high levels of ability. This indicator cannot be calculated for Kentucky or the nation in 2002. With data drawn from a measure like the Collegiate Results Survey (CRS), it would be calculated as the number of respondents reporting a high level of ability as a proportion of the total number of survey respondents. (Indicator Weight = 5%)

Quality of Educational Outcomes (Weight = 30%)

  • Performance of college graduates on tests of advanced abilities. This indicator cannot be calculated for Kentucky or the nation in 2002 because there is no suitable assessment instrument. The measure would be calculated by comparing the mean scores of Kentucky graduates to those of the nation's graduates, and scores would be broken down by area of performance. For example, they might be reported in terms of key abilities (such as communication and problem solving) identified by the National Education Goals Panel. (Indicator Weight = 20%)

  • Performance of college graduates on tests for advanced practice. Calculated as the aggregated mean scores or pass rates of Kentucky graduates on licensure and admissions examinations, compared to those of the nation's graduates. Ideally, these indicators would be reported separately for (a) licensure examinations at the four-year college level, (b) licensure examinations at the two-year college level, and (c) graduate admissions tests. (Indicator Weight = 10%)

Good Practices in Undergraduate Education (Weight = 15%)

  • Good practices in four-year colleges. Calculated as the aggregated benchmark scores of all respondents to the NSSE in Kentucky, compared to those of the nation. Indicators are reported separately for each of the five NSSE benchmarks, and are weighted by the number of full-time-equivalent (FTE) students enrolled at the participating institutions. (Indicator Weight = 7.5%)
  • Good practices in two-year colleges. This indicator cannot be calculated for Kentucky or the nation in 2002. If CCSSE data were available, the indicator would be calculated as the aggregated benchmark scores of all respondents in Kentucky, compared to those of the nation. Indicators would be reported separately for each of the CCSSE benchmarks, and would be weighted by the FTE enrollment of the participating institutions. (Indicator Weight = 7.5%)

Creating and Reading a State Profile

The results of this exercise are shown in the accompanying table. Although the data are unsatisfactory and incomplete, the table illustrates how a comparative profile of state performance with respect to educational capital might be constructed. Even this barest of profiles suggests several themes for policymakers to consider:
  • Verbal literacy levels for Kentucky's college-educated residents are better than average. The abilities of high-school educated Kentuckians are well above average in these areas, especially in younger age groups-possibly reflecting the impact of recent K-12 reform programs. But Kentucky remains well below the nation in quantitative literacy levels despite these reforms.

  • Kentucky's higher education outcomes and good practices are only average, perhaps a bit below average. Colleges and universities in Kentucky do not appear to have kept pace with K-12 reform.

  • Kentucky's educational institutions contribute more to the vocational/professional dimension of the state's educational capital than to the more "academic" dimension of preparing graduates for further study. Kentucky is well below the national average with respect to the absolute numbers of students taking competitive admissions examinations and, despite fewer test-takers, is also somewhat below average in performance.

back to top