University of Toronto Law School Faculty Blog

Balanced Climate Policy, Made in Canada?

In a September 25 poll on its website, the Globe and Mail asked Canadians: “In terms of global warming, what approach do you favour? Strict adherence to the Kyoto Protocol? The 'more flexible' approach suggested by Prime Minister Harper? A compromise between those two positions? No action is needed because global warming is not happening?” The about 86000 respondents were almost equally divided between the first three options, with only 5 % opting for “no action.”

But are politicians and media providing the information that would allow Canadians to make these judgments?  Kyoto-bashing has become something of a national sport, a sport that Canadian media have fallen for hook, line and sinker.  The same goes for political slogans touting the need for a “made in Canada solution” or a “balanced approach” to climate change.  Why indeed favour a “flawed treaty”? And why resist such reasonableness as home-made balance?

Supreme Court of Canada Justices are Unpredictable -- Mostly, Part II

This is a follow-up to the previous post regarding the paper Andrew Green and I recently posted on SSRN.  The point of this post is to elaborate on the motivation for the paper and summarize our results.

The debate: There is an active debate surrounding the desirability of a more open appointments process for Supreme Court of Canada justices. On the one side of the debate are those militating for more openness, transparency, and public input into the selection of justices for the Court. The argument is that since the decisions of the Court often have such wide-ranging social and economic effects—often not reversible by Parliament or legislatures (other than temporarily through the use of the notwithstanding clause)—there should be a role for democratic input into the selection of justices. Typically the envisioned opening of the process would involve Parliamentary scrutiny and vetting of nominees, with the aim of exposing and testing the policy inclinations of would-be justices. On the other side of the debate are those who fear that Parliamentary involvement in the selection of justices would threaten judicial independence and politicize the selection of justices in ways that the process has avoided in the past. Further, opponents of a Parliamentary process fear that strong candidates that would otherwise be willing to consider an appointment might refuse in order to avoid putting their personal lives before a national audience and to avoid the unpleasantness of aggressive questioning by Parliamentarians who are keen to score political points.

The common assumptions: One of the central assumptions underlying the debate about appointments to the Supreme Court of Canada is that it is possible to know “the policy preferences” or “ideology” of a prospective justice before they are appointed. After all, if Prime Ministers cannot easily determine ex ante how a particular justice is likely to vote in politically divisive appeals, then it would seem that the hopes of those in the first camp of exposing and testing nominees’ political preferences are fanciful. A further assumption is that justices come “pre-loaded” with a certain policy orientation, and that this policy orientation does not (or is unlikely to) change over time.  In other words, there is an assumption that justices will have unwavering, constant policy preferences. This assumption is important because even if a justice’s policy preferences can be known at the time of appointment, this information may be of only short-run relevance if the views of justices are subject to continuous revision, refinement and change.

Supreme Court of Canada Justices are Unpredictable -- Mostly

Andrew Green and I have posted a working paper, Policy Preference Change and Appointments to the Supreme Court of Canada to SSRN.  Here's the abstract:

As Canadian courts have taken on a greater role in reviewing government action following the advent of the Charter of Rights and Freedoms, judges have been increasingly criticized for making social policy. The criticism of judicial decision-making has brought the judicial appointment process under scrutiny, particularly the process for appointing justices to the Supreme Court of Canada. Supreme Court justices are appointed by the Prime Minister through a process which has in the past not been open to public scrutiny. There have been increasing calls to reform the process, possibly by instituting a U.S. style public hearing process prior to appointments. The assumptions underlying the debate are that Prime Ministers can predict the voting of their appointees, that they appoint justices whose political preferences accord with their own and that these judges vote in a predictable pattern over time.

Unlike in the U.S., there has not been a considerable amount of empirical work undertaken on decision-making by justices of the Supreme Court of Canada. This paper examines Supreme Court of Canada decision-making in the post-Charter era. We examine how the policy preferences of the justices on the Supreme Court of Canada between 1982 and 2004 shifted (if at all) during their time on the Court, focusing particular attention on the behaviour of the justices immediately following their appointment. We use two methodologies to assess preference change among the justices of the Supreme Court of Canada. One is a direct methodology based on the observation of judicial votes in various areas of law. The other is an indirect methodology, which uses Bayesian inference and a Markov Chain Monte Carlo methodology to uncover the latent policy preferences of the justices based on a spatial item-response theory model. In undertaking our analysis, we first assume that judges have constant preferences over their tenures, and then relax this assumption and assess the consequences. Our results suggest that the policy preferences of justices shift over time and the analysis has implications both for the debate surrounding the Canadian appointments process and for models of judicial decision-making.

Improving the Rankings

Ideally rankings should be based on aspects of law school performance that can garner wide acceptance and cannot easily be manipulated.  That is, for a law school to improve its position in the rankings, it should have to actually improve its performance as an institution.  The two main activities of a law school are research and teaching.  Therefore, in measuring the performance of law schools in order to rank them, the goal should be to measure research contributions and teaching effectiveness in an unbiased way.  This is not easy to do.

First, some ideas about improving the measure of "faculty quality." The Maclean's ranking measured "faculty quality" by determining the per capita number of citations to faculty work in the 33 Canadian law reviews found in the Quicklaw journals database.  This was intended to measure the "impact" of the research produced by members of each faculty.  One limitation of this measure is that it ignores important non-Canadian journals.  One way to improve this measure, then, would be to add a list of such journals to the 33 Canadian ones that are already included in the measure.  A (possibly) relatively uncontroversial way of compiling this list of "other journals" would be to include any electronically search-able academic journal that has published at least X articles (say, at least three in order to avoid the especially thin part of the distribution) in it by current faculty members at Canadian law schools.  This would probably not dramatically increase the number of academic journals included in the measure, but would result in a better targeted list of journals to scour for citations.  For example, it would pick up journals like the The New England Journal of Medicine, Oxford Journal of Legal Studies, Philosophy and Public Affairs, American Law and Economics Review, Modern Law Review, Law Quarterly Review, The Journal of Legal Studies, and an undetermined number of others.  I would also exclude self-citations from the measure, since including self-citation rewards self-promotion rather than true scholarly impact.  It might also be desirable to scale citations according to the quality of the journal that the citations appear in, though this would probably be very difficult to do without arousing considerable controversy (for this reason, I would probably avoid trying to do so).  My guess is that this extended list of academic journals would not dramatically affect the ordering of the law schools with respect to faculty quality, but it would at least counter the incentive that measuring only Canadian journals would have on decisions about where to publish one's research (and what to write about).

Why Rankings at All?

In two previous posts, I discussed the recent ranking of Canadian law schools in Maclean's magazine.  Before I make a post with my suggestions for improving the ranking methodology, however, I think it would be helpful to establish the spirit in which I will make these suggestions.  To set the stage, let me explain what I think are some of the legitimate roles of ranking systems.

The appropriate motivation behind constructing a ranking of anything, including law schools, I take it, is to systematically collect and analyze information about an array of elements in a way that provides some output that is meaningful for the users of the ranking.  Various ranking systems with meaningful output exist in other contexts.  ELO ratings in chess provide information about an individual's playing ability (and also information about a chess program's playing ability) by summarizing the relative performance of players in a more telling way that a pure Win-Loss-Draw tabulation.  It can do this because the ELO ranking system takes into account the playing ability of each player's opponents.  The World Golf Rankings provide rankings of golfers based on a player's performance over the past two years in certain golf tournaments.  For the golf rankings the points awarded for any given result is based on a combination of the strength of the tournament's field and its proximity in time (more remote results are given less weight).  Ranking systems are also in use in other sporting contexts, such as tennis, squash, (indeed most, perhaps all, racquet sports), bridge, etc.  There is some consensus, I believe, that in each context in which such rankings are used they are imperfect in that they are not perfect indicators of future performance.  But it does not follow from this that they are useless and should be abandoned.  To the extent that they are well-designed, and collect and summarize information that can be useful in predicting future performance, they are useful.  In a similar way, in the law school context a ranking methodology can be worthwhile if it has the practical effect of consolidating, summarizing, and brokering information about a range of institutions in a way that reduces noise, increases the availability of meaningful information about an institution's results, and predicts (albeit necessarily imperfectly) future results.

Maclean's Law School Rankings: Observations on Methodology

As mentioned in the previous post, the Maclean's ranking of Canadian law schools uses four elements to produce an overall ranking of Canadian law schools.  This ranking is based on:

• Faculty quality (50%)
• Student (more aptly graduate) quality (total of 50%)
  • elite firm hiring (25%)
  • national reach (15%)
  • Supreme Court Clerkship hiring (10%)

At this general level, it is difficult to quibble with the concept of assessing law schools on the basis of performance in faculty research and in teaching outputs.  Taking a closer look at how each of these four components is operationalized and measured, however, suggests a number of important limitations to the rankings.

Maclean's Law School Rankings

The September 24th edition of Maclean's magazine is going to have a feature story in which Canadian law schools are ranked according to a methodology that is "simple, transparent and relies entirely on public data."

Professor Brian Leiter, well known for his ranking of US law schools, was retained by Maclean's to assist in the design of the ranking.  The magazine's website has a brief discussion of the methodology, but the basic story is that 50% of the ranking is given to faculty research quality (as measured by citations of work in Canadian law reviews) and 50% of the ranking is based on student quality.  The student quality measure, in turn, is an amalgam of three measures: (i) "elite firm hiring"; (ii) "national reach" (15%); and (iii) "Supreme Court hiring" (10%).  The methodology seems reasonably sensible, albeit somewhat arbitrary (as any method of ranking law schools is apt to be). 

The key requirements of any such ranking of law schools to my mind are two-fold.  The first surrounds whether the ranking is attempting conceptually to capture the most important and relevant measures of law school performance (in all its facets).  It would not be surprising if reasonable people had different views as to what the "most important and relevant measures of law school performance" are, and I don't purport to know what they ought necessarily to be, though faculty quality and the success of graduates do appear to be quite reasonable selections at a conceptual level.  The second key requirement is whether, as the conceptual measures are operationalized, they are in fact capturing what they are purporting to capture.  I will have more to say about how well the methodology adopted for this ranking satisfies this second requirement in a post soon to follow this one (this evening or tomorrow morning).

The magazine has not provided on its website information on the overall rankings, but has provided information on each of four components going into the overall ranking.  Of course, some rudimentary Excel skills can rapidly convert these sub-indices into an overall ranking given the reported weights.

Without further ado, here are the results for Canada's common law schools.  (Caveat emptor: these are my own hasty calculations.)

UPDATE: I've been told by a reliable source that while these calculations are close, they are not spot on, because the overall ranking uses z-scores for each of the components rather than a weighted average of the ordinal rankings in each.

UPDATE 2: Here are the actually reported rankings: