Choosing Elites by Robert Klitgaard

Forty years ago Robert Klitgaard was a Harvard golden boy. Still in his mid-thirties, with three Harvard degrees (A.B., Philosophy; M.P.P., Public Policy; and Ph.D., Public Policy), he was an associate professor at the Kennedy School of Government while spending half his time as Special Assistant to Harvard President Derek Bok. He had also served as Admissions Chairman for Harvard’s Public Policy Program and published extensively. He appeared to be on a fast track leading to the presidency of, if not Harvard, at least Yale, Princeton, or Stanford.

Then he wrote a book.

The book was Choosing Elites, about “[s]electing the ‘best and the brightest’ at top universities and elsewhere.” It is not anexposé, critique, or call for reform. The primary target audience appears to have been admissions officers and others involved in setting admissions policies for highly selective colleges and universities, but Klitgaard believed that some of his insights could also be helpful “to personnel selection in business and government, screening systems in medicine and elsewhere, the design of contests, and even to professional sports teams making draft choices.”

Choosing Elites is well-written, thoughtful, lucid, deeply-researched, balanced, and scholarly without being pedantic. Klitgaard is the opposite of tendentious. In his review of relevant studies, he not only digs into the methodological details but sometimes reanalyzes the data upon which they were based. He understands the methods and limitations of statistical analysis. (Despite his formal academic credentials, Klitgaard is often described as an economist, and has taught economics) As a Harvard admissions insider, he not only had access to admissions information for the Kennedy School and all of the other Harvard programs, but he could and did call up his counterparts at other selective schools and institutions and got non-public information and candid assessments. The book includes two useful technical appendices, Evidence on Academic Prediction at the Right Tail and Academic Variables and Later-Life Success.

Klitgaard does not confine himself to the technical literature, but intersperses apt observations from the likes of Confucius, Peter Drucker, Learned Hand, Nietzsche, Plato, Derek de Solla Price, John Rawls, Joseph Stiglitz, and Steven Weinberg. Whether or not you agree with his conclusions, there is a wealth of interesting information in this book.

Klitgaard’s fundamental message is that selective universities and other institutions that choose elites from the right tail of the normal distribution do not think hard enough about what they are doing, and many of their selection criteria are not adequately justified. He finds that admissions policies “are not derived in even a moderately explicit way from a careful specification of each institution’s differing objectives,” that “the objectives of selection are remarkably vague” and vary from year to year, and that “[i]t is remarkable how little systematic information is used by admissions people in making their judgments.”

I say all this as a former admissions chairman. It is a provocative exaggeration to say that we make these important decisions without clear objectives, without relevant feedback, and without accountability to students, faculty, and alumni. But it is almost that way. Indeed that it is that way may be well and good. I do not wish to prejudge appropriateness.

Klitgaard does not tell selective schools what their admissions policies should be; he prefers to construct intellectual frameworks they could use to answer that question for themselves. For example, here is the framework concerning the representation of groups:

Of course, this sort of explicit, rigorous process is not what admissions committees traditionally do or have ever done. Admissions offices are not staffed with economists. They are unwilling or incapable of thinking that hard about what they do. But it is difficult to deny that the framework approach makes a lot of sense.

Although his actual recommendations are largely limited to such frameworks, Klitgaard reaches some conclusions that he finds “disheartening” and “unsettling.” What dismays him the most is that:

This book, and this was a surprise to me, has ended up supporting an admissions system giving more weight to “academic merit” at selective universities like Harvard. This tentative conclusion depends of course on many value judgments, but its primary source was factual: Given the current state-of-the-art of prediction at the right tail, selective universities will do better achieving their objectives by choosing the academically ablest students [as predicted by test scores and grades], with appropriate allowance for the representation of groups. We simply cannot predict much of interest with the other intuitively pleasing criteria now available.

By “intuitively pleasing criteria” he means things like interviews, letters of recommendation, essays, extracurriculars, and “other indicators of ‘leadership,’ ‘character,’ and ‘personality.’ . . .” He does not deny that such criteria would be useful if they were all that admissions committees had to rely on; he simply means that they add little useful information, and to the extent that their consideration overrides test scores and grades they have significant downsides. What Klitgaard found was that Harvard was incurring a very substantial academic cost in order to admit the many athletes, legacies, members of minority groups and other applicants who were appealing to admissions officers but would not have been offered admittance on the basis of academic merit alone. He does not say this is necessarily a bad tradeoff, just that it is a big tradeoff of quantifiable academic costs for very uncertain and questionable benefits. He estimates that:

[h]olding constant high school grades, a 200-point increase in SAT V+M corresponds to an increase in grades at a selective college of about four-tenths of a standard deviation. This is roughly the difference between a student at the 33rd percentile and another student at the 50th percentile of the college class.

How much does Klitgaard’s central finding matter? That depends on how important you believe it is for elite schools to maximize the transmission of knowledge. Klitgaard discusses a study by Harvard economist Elisabeth Allison which vividly illustrates the significance in the classroom of academic ability, as defined by standardized tests:

Looking at Harvard’s survey course in economics and using a complicated set of simultaneous equations, she estimated how a host of variables - what students did with their time, students’ tastes and abilities, teachers’ characteristics, and the way the course was taught in different sections (pass/fail, programmed instruction, and so forth) - affected how much economics students learned and their satisfaction with the course. How much they learned was estimated via an objective examination given at the end of each semester. With all these variables ingeniously and extensively measured over a number of years, Allison made the following discoveries:

With other variables held constant, a measure of “academic ability” that combined high-school records and SAT test scores was about ten times as important as a measure of “student effort” (study time and so forth), in terms of how much was learned.
Other things equal, a hundred point increase in both the SAT verbal and math scores made a slightly greater difference as to how much more students learned than did the difference between the best and worst instructors in the course.
Students’ ratings of the competence and personability of their teachers had absolutely no relationship with how much the students learned. But the teachers’ grades in graduate school did.
A twenty-five point increase in both SAT verbal and math scores had about the same effect on how much economics was learned as an increase in study time of ten hours per week for this course.[footnote omitted]

In other words, brains matter. A lot.

Turning to the representation of groups, Klitgaard observes that:

An admissions policy that ignored the issue of group representation would be subject to grave criticisms. So would a book on choosing elites. And yet, as I pursued this research at Harvard and other American universities, several colleagues and friends suggested that such issues were best left alone. They wondered whether it would be unpleasant or risky or even wrong to study group differences, bias, and representation. Some thought that studying bias and the performance of minorities could heighten racial sensitivities, with negative consequences. Minorities might feel hurt if they thought that as a group their test scores and grades and academic performance were being studied. Admissions officers and university administrators might worry that the results of an investigation of bias would cast them in a bad light. And those who studied such matters might find their integrity questioned.

The concerns of the “colleagues and friends” were, of course, well-founded. But Klitgaard felt that they were outweighed by the argument that issues of group representation and bias in testing deserved careful study.

Most universities now have strong affirmative action policies. To implement these, sophisticated understanding is needed of the characteristics of the potential applicant pool, the predictive accuracy of admissions criteria for different groups, and the benefits and costs of an ethnically diverse student body. In choosing elites, we need to think hard about how much affirmative action is desirable, not simply whether affirmative action should exist.

Klitgaard makes three major findings about the representation of groups. First, that “[a]t the right tail of academic qualifications, there are surprisingly large differences in the performance of various ethnic groups.” Second, that:

Differences in scores cannot be attributed to predictive bias in the tests. Indeed, predictions made using test scores and high-school grades actually overstate the later performance of blacks relative to whites. Compared to whites with the same test scores, blacks on average underperform in college, in graduate schools, and on some measures of job performance. The degree of this underperformance is from one-third to two-thirds of a standard deviation at typical right-tail institutions.

(on this issue, Klitgaard was doubling-down on an earlier internal report that had generated controversy a few years earlier when it was leaked to the Harvard Crimson)

Finally, that:

Even across institutions with identical views about the marginal benefits of increased representation of blacks, there is no one “right” amount of affirmative action. The marginal costs of representation will differ across institutions, depending in part on how selective they are and how well they can predict the later academic performance of whites and blacks. In general, the more selective the institution and the better its prediction of later performance, the higher the costs of increased representation will be.

Klitgaard ends his discussion of the representation of groups with an appeal to a reasoned approach.

There is no pretense here of a definitive assessment of the complex issue of group representation. Nor, as I have emphasized throughout, does my focus on issues of fact countermand the view that the issue is driven primarily by questions of ethics and politics. But here, as with other difficult issues, we need more facts and analyses, not fewer. Universities in particular should try to illuminate the debate over racial representation in choosing elites, not cloud things over with rhetoric and white lies. As elsewhere, we must hope that thinking harder is part of the solution.

The universities, of course, decided in their collective wisdom that thinking harder about group representation was too hard, and have continued the policy of clouding things over.

In his final chapter, Concluding Remarks, Klitgaard hazards these general propositions:

In highly selective universities, test scores and previous grades are quite helpful in assembling a class that on the whole will perform well academically (citation omitted).
At the right tail, test scores and grades are much less successful in forecasting various kinds of success in later life (citation omitted).
Other information now used in selection - such as essays, recommendations, interviews, and biographical information - do not add much of importance to the prediction of various kinds of later success (citation omitted).
Using personality measures and other psychological information in selection at the right tail would probably not help - in part because there is no evidence that they predict later performance, and in part because an applicant can easily be coached to give the right answers. . . .

Klitgaard’s findings and conclusions seem quite damning of the actual practices of selective schools, but he emphasizes that he is not saying that the schools are getting their admissions decisions wrong, and need to change the composition of their student bodies. But although Klitgaard does not lay out a reform agenda, he does urge schools and their admissions committees to face facts about what they are doing.

. . . if we are in charge of choosing an elite, we have more than one responsibility. We do have the task of providing information about the implications of alternative decisions and policies, which has been the main aim of this book. But we also have to create legitimacy for decisions via a fair process of choice. We have to allocate responsibility for decisions at the appropriate levels and positions of our institutions. We must create opportunities for later review of the decisions made - developing more information, checking on implementation, and so forth. All these dimensions should be included among an institution’s criteria for a successful selection policy.

Choosing Elites ends on a muted, slightly discouraged note.

The complications of this topic - not just the uncertainties of current knowledge, but the psychological and statistical complexities involved - create a certain psychological barrier to the use of analysis and empirical data. We may be reluctant to connect statistical distributions and predictions to the real candidates encountered, however sketchily, in selection files and interviews. We may even resist putting subjective judgments to the test - finding out whether pet theories and intuitive hunches serve better than the alternatives.

In closing, Klitgaard, rather than calling for any specific changes in admissions policies, expresses the hope “that new research will lead us to question productively our values and our theories about selection at the right tail, and eventually to improve policies for choosing elites.”

Klitgaard’s modest hopes were disappointed. The choosers of elites were not interested in his systematic approach, then or since. They did not want to question their values and theories about selection. Klitgaard was swimming against the cultural tide. Choosing Elites was received respectfully, but it did not make much of a splash. The New York Times reviewer, an education professor at SUNY-Buffalo, opined that those involved with admissions should forget about trying to predict intellectual success and think about “kind - kind of student, kind of thought, kind of college, kind of social belief that will improve the quality of moral as well as intellectual life.” Whatever the reviewer meant, it was obviously a rejection of Klitgaard’s approach.

The University of Minnesota psychologist Thomas Bouchard, of twin studies fame, wrote a positive review in, of all places, the law journal Constitutional Commentary. He opined that Klitgaard’s “conclusions were sound and well supported by the available evidence.”

With regard to the role of human judgment in the selection process, Bouchard adds:

The problem of clinical versus statistical prediction is an old issue in psychology. In lay terms, the question is whether one can predict a candidate's performance (as a student, a professor, a lawyer, or what-have-you) better after interviewing him, or some other subjective procedure, than by rigid statistical methods. Most people believe that they can, but the evidence suggests otherwise. The fundamental problems were systematically organized and evaluated in 1955 by Paul Meehl.[footnote omitted] A large body of evidence, encompassing many new problems of prediction, has accumulated on this topic and it all points toward the same conclusion: human judges are susceptible to multiple sources of error rendering it improbable that they will predict well and certain that they will do less well than statistical procedures. Human judgment in the form of individualized evaluations is more likely to lead to results that are arbitrary rather than fair or valid.(footnote omitted)

Unlike in the case of The Bell Curve ten years later, there was no concerted effort to discredit Choosing Elites, perhaps because it never threatened to have a significant impact on the admissions policies of highly selective schools. Forty years later, as shown by pre-trial discovery in the Harvard affirmative action case, little had changed in the Harvard undergraduate admissions process, other than the rise of Asian students as an important group. Has the relationship between grades/test scores and academic performance changed? There is no reason to think so, but it is undoubtedly more difficult to discern. Grade inflation at the high school and college level and the concomitant ceiling effects, together with the disappearance of rigorous required courses that all students must take, have made grades a much less reliable indicator of academic achievement. At the same time, the recentering of the SAT created a lower ceiling that makes it more difficult to use the test to distinguish among applicants at the far right tail of the distribution. A SAT verbal or math score of 800 doesn’t mean what it used to.

Since the publication of Choosing Elites, there have been serious studies of affirmative action - e.g., The Shape of the River by Bowen & Bok (for) and Mismatch, by Sander & Taylor (against) - and many books purporting to pull back the curtain on college admissions. No one has probed as deeply into selection at the right tail as Klitgaard, but he has wisely, in view of his academic career, stayed out of the college admissions debate.

Robert Klitgaard never became president of Harvard, and I can’t help but believe that his authorship of Choosing Elites had something to do with it. On the other hand, the presidency of Harvard no longer seems to have the prestige and importance it once had. And Klitgaard ended up having an impressive and productive career, worthy of his gifts. He left Harvard after Choosing Elites was published and became one of the most prominent and respected experts on governmental corruption and economic development in poor countries. He served as Dean of the RAND Graduate School and President of Claremont Graduate University, where he continues today as a University Professor. He is still highly productive, churning out books and articles, seemingly at an accelerating pace. And he has had a lasting marriage to a beautiful woman. 2 One wonders if he ever made any effort to bring analytical models and empirical research to bear on the admissions processes of RAND and Claremont, and if so whether he enjoyed any success. Perhaps next year, on the 40th anniversary of its publication, he will favor us with his reflections on how his Choosing Elites findings and conclusions look to him today.