Multijurisdictional Right to Counsel
Multijurisdictional Right to Counsel
Home - Multijurisdictional Right to Counsel About Us - Multijurisdictional Right to Counsel Membership - Multijurisdictional Right to Counsel Advocacy In Progress - Multijurisdictional Right to Counsel MJP Developments - Multijurisdictional Right to Counsel Contact Us - Multijurisdictional Right to Counsel Online Resources - Multijurisdictional Right to Counsel

Why Was Former Stanford Law School Dean Kathleen Sullivan Failed on the California Experienced Attorney Bar Exam?

Notwithstanding that the California licensing test for experienced attorneys is graded in an assembly line and not a valid or reliable test, and two out of three experienced attorneys are routinely failed on the July bar exam, (see previous post), the question naturally arises: How could former Stanford Law School Dean Kathleen Sullivan, the author of a textbook on Constitutional Law and one of the most skilled advocates in America, be failed on this entry level licensing test? The answer lies in what scientists have termed “inattentional blindness.” A taped study at Harvard University demonstrates “inattentional blindness.” There were two teams of basketball players, one wearing blue and one white, throwing the ball to each other. Participants were asked to count the passes from team members to each other. A man in a gorilla costume walked into the middle of the action, brushing shoulders with the players, turned directly toward the camera, beat his chest, and slowly walked off the Court. The study showed that more than fifty percent of the participants in the study were so absorbed in counting the passes they did not see the gorilla in their midst. Afterwards, when the participants watched the tape again, they were so surprised when the gorilla was pointed out to them that many of them accused the experimenters of having doctored the tape between viewings.1 The bar exam graders are paying attention to entry level signs, and experienced attorneys, like former Stanford Law School Dean Kathleen Sullivan who was failed on the exam, is a gorilla in their midst.

Kathleen Sullivan

Permanent Link: Why Was Former Stanford Law School Dean Kathleen Sullivan Failed on the California Experienced Attorney Bar Exam?

Bar Examiners on Steroids: Five Nationally Respected Testing Experts Conclude the California Experienced Attorney Licensing Test Fails to Satisfy Testing Standards

Five nationally respected testing experts have concluded that California’s 100% subjective high-stakes licensing tests given to experienced sister-state attorneys does not meet testing Standards.
The ABA, in addition to endorsing admission on motion, adopted a first-time bar passage rate of at least 75%, or its member schools risk losing their accreditation.1 The first-time pass percentage pass rate on the July 2008 bar exam was 83%.2 The chart below reflects the statistics on the California July experienced attorneys’ examination, and the experienced attorney pass percentage in the minority of states still requiring experienced attorneys to reinvent the wheel, and take another bar exam.3
*             2004              2005              2006
Jurisd    Passing           Passing          Passing
*        Percentage       Percentage    Percentage

CA        39%                  28%                36%

GA        87%                  87%                87%

ME        90%                  100%              80%

MD       92%                   88%               96%

MS       100%                 100%              67%

RI         80%                  100%              94%

UT        100%                 89%               50%

total      57%                  52%                59%

The disparate impact of this licensing exam raises red flags that warrant further investigation. Statistics, in general, dictate that most populations will have a Bell Curve shaped distribution, with 2/3 of the population within one standard deviation from the mean. According to California State licensing officials, two out of three attorneys already licensed in other states, are threats to the public and cannot be safely trusted to practice law.

There are standard statistical indices for the reliability of test scores. The most basic of these is the standard error of measurement. A reliability coefficient can be defined in terms of the average magnitude of the standard error. A fairly high reliability (above 0.8; preferably above 0.9) is expected for testing programs that are used to make high-stakes decisions about individuals.4 For example, if two graders agree on the score, the correlation coefficient is 1.00 and if there is no agreement the correlation coefficient is zero. Bar examinations are high-stakes licensing examinations because they have serious consequences.5

The State Bar of California prepares a report on each bar exam for the California Supreme Court. Standard 14.15 provides that “estimates of the reliability of test-based credentialing decisions should be provided.” In light of Standard 14.15, deeply buried so as to be unnoticeable, the State Bar provides an estimate of the reliability by way of a correlation coefficient. That is, the degree by which the graders on the subjective test sections agree with themselves. The measurement error is another way of referencing the correlation coefficient. The Report(s) to the California Supreme Court on the California Bar Examination documents the reader correlation on the 100% subjective eight-question subjective tests given to experienced attorneys as follows:

February 2001 reader correlation .41
July 2001 reader correlation .48
February 2002 reader correlation .38
July 2002 reader correlation .40
February 2003 reader correlation .48
February 2004 reader correlation .39
July 2004 reader correlation .41

These experienced attorney reliability results confirm Dr. Norman’s conclusion that getting bar graders to agree on subjective scores is virtually impossible.

First, Dr. Phillip L. Ackerman is a Professor of Psychology at Georgia Institute of Technology; the Editor, Journal of Experimental Psychology: Applied; a Fellow of the American Psychological Association and a member of the American Educational Research Association and the National Council on Measurement in Education (these are the three organizations that generate the Standards on Psychological and Educational Testing). Dr. Ackerman wrote the chapter on testing in K. Anders Ericsson, Ed., The Cambridge Handbook of Expertise and Expert Performance, supra, and he is one of the leading testing experts in the world.6 Dr. Ackerman has reviewed the State Bar’s Report to the California Supreme Court on the California Bar Examination and other material he deemed necessary to form an expert opinion.

Dr. Ackerman’s professional opinion is that the Attorney’s Examination for experienced sister-state attorneys fails to meet the Standards for Educational and Psychological Testing. Multiple Standards have not been met. More particularly, Dr. Ackerman declares, under oath:
“The scores on the Attorney’s Examination are determined in a manner that is not consistent with professional standards. The reliability of the test scoring procedures fails to reach a level that would be acceptable for high-stakes testing. (Specifically, inter-rater agreement is quite low, a correlation of .48 between raters indicates only 23% shared variance among ratings; source: Klein & Bolus; Gansk & Associates 2003.) An acceptable level of reliability for such high-stakes testing would be shared variance in the neighborhood of 70% or higher (corresponding to reliability of about .84 or higher). (Emphasis added) (Exhibit A ¶ 4c)

When the goal of inter-rater reliability is preferably in the range of .8 to .9 as noted by Dr. Kane, and the inter-rater reliability of the California Attorney’s Examination is consistently reported to be below .5, there can be little doubt that the reliability of the decisions made on the basis of the scores is extremely low, and not acceptable. (Exhibit A ¶ 9)

Dr. Ackerman further concludes the “Attorney’s Examination lacks content-related validity” (Exhibit A ¶ 4a), and it “has never been demonstrated to have criterion related validation, (Emphasis added) in terms of evaluating the scores on the test and comparing them to performance of practicing attorneys. (Exhibit A ¶ 4b).

Second, Dr. Gary H. McClelland, a professor at the University of Colorado at Boulder is also an expert on statistics and measurement. Dr. McClelland previously studied the Colorado bar examination and based on that study wrote “Assessing Bias in Professional Licensing Examinations by Checking Internal Consistency,” 9 Law and Human Behavior, No. 3, p. 305 (1985). Dr. McClelland declares, under oath:

I have reviewed Dr. Phillip L. Ackerman’s “Evaluation of the Psychometric Adequacy of the California Attorney’s Examination” dated February 15, 2008, and generally agree with it. Dr. Ackerman is a credible psychometrician as well. In my opinion, the lack of an explicit equating procedure for the Essay and Performance Test sections is a fatal flaw. The degree of inter-rater agreement is dreadful. I do not believe any scientist would ever publish data based on such low inter-rater agreement. (Emphasis added).

Third, Dr. Susan Case, the Director of Testing for the NCBE, avows that non-multiple choice format tests, such as essay and performance tests “because of their limitations, such as low reliability, lack of anonymity, and lack of standardization, should not be used in isolation.” See Susan M. Case, “Licensure In My Ideal World,” The Bar Examiner, p. 27 November 2005. The California Bar exam for experienced attorneys is a 100% subjective test used in isolation.

Fourth, the NCBE’s Dr. Kane opines, “If it were feasible to evaluate performance in practice directly, this would be the preferred approach.” Michael T. Kane, “The Role of Licensure Tests,” The Bar Examiner, p. 34 (February 2005). According to the ABA and the NCBE, nationally respected peer groups, the preferred approach is to evaluate practice directly based on the attorney’s experience. Dr. Kane, as noted above, states the industry standard for reliability is preferably .9 for bar examinations.

Fifth, Dr. Stephen P. Klein prepares the report for the California Supreme Court on each bar exam noted. Dr. Klein is paid handsomely to provide the California Supreme Court with full and fair disclosure. Dr. Klein in other writings, however, emphasizes the danger caused by using 100% subjective high-stakes tests in isolation. See Stephen P. Klein, “What Do Test Scores in Texas Tell Us?” (Published 2000 by RAND) Dr. Klein admits:
“Our research results illustrate the danger of relying on statewide test scores as the sole measure of student achievement when these scores are used to make high-stakes decisions about teachers and schools as well as students. We anticipate that our findings will be of interest to local, state, and national educational policymakers, legislators, educators, and fellow researchers and measurement specialists.” 7

This Dr. Klein evidentiary admission proves the California bar exam for experienced attorneys is fundamentally flawed because it is used as the sole measure of experienced attorney competence.

Dr. Klein also admits, “While many bar exam graders believe they can recognize a passing answer when they see one, there is strong empirical evidence to the contrary.” See Stephen P. Klein, “Essay Grading: Fictions, Facts and Forecasts,” The Bar Examiner p. 23, 25 (August 1985).
Clear and compelling evidence shows the California bar exam for experienced attorneys, the putative gold standard, fails to satisfy testing Standards.

1 “ABA Votes on Bar Passage Journal” Daily Journal February 13, 2008 p. 3.
2 Los Angeles Daily Journal November 25, 2008 p.1
3 From the NCBE web page http://www.ncbex.org/bar-admissions/stats/
4 Id. at p 9.
5 Id. at 9. 
6 Over the past 15 years, Dr. Ackerman has published 12 reviews in the Mental Measurements Yearbook (which is generally regarded as the “bible” for critical reviews of commercial, educational, psychological, and organizational tests). Over the past 25 years, he has consulted on educational and occupational testing for the following organizations: U.S. Air Force, U.S. Army, U.S. Navy, Personnel Research and Development Center, U.S. Department of Education, Minnesota Air Traffic Control Center (FAA), The College Board, Educational Testing Service (ETS), and General Motors. 

7 http://www.rand.org/pubs/issue_papers/IP202/index2.html;

Permanent Link: Bar Examiners on Steroids: Five Nationally Respected Testing Experts Conclude the California Experienced Attorney Licensing Test Fails to Satisfy Testing Standards


Canada has made a concerted effort to provide its lawyers with inter-province mobility, allowing them to practice law outside of their legal training and licensing jurisdiction. Specifically, Canada enacted the National Mobility Agreement and Territorial Mobility Agreement. Under these rules, an attorney licensed in one Canadian province may practice law in another jurisdiction, as long as he or she is entitled to practice in their home jurisdiction (i.e. they are in good standing). Further, an applicant attempting to practice in another jurisdiction need not pass that jurisdiction’s legal licensing examinations; they must simply certify that they understand the materials and rules reasonably required in that to practice in that jurisdiction. These rules allow Canadian attorneys to freely follow their clients and practice law where they so please. In Canada, one bar examination is sufficient.



The European Union has concluded that one bar exam is more than enough. In the EU there are 40 countries, one million lawyers, and 23 official languages. In stark contrast to the American system, there is a unified goal of homogeny and mutual reciprocation amongst the European Union countries. Entry into the practice of law is regulated to be in the best interest of the citizens represented by attorneys. A lawyer licensed in one country can open an office temporarily or permanently in any country in the EU. Once practicing for three years in that country he becomes a member of his new country’s profession in addition to his initial licensing country. Further, law students and other people in the middle of legal training may transfer to other EU countries to compete their legal training and licensing. This allows people engaged in legal training to choose their country of practice, without fear of being pigeonholed in their “home” country.



Copyright © National Association for the Advancement of Multijurisdiction Practice. All Rights Reserved.
MJP, Multijurisdiction Practice, Attorneys, Lawyers, Right to Travel, First Amendment Rights, Speech, Advocacy, Association, Access to Courts, Speaker Discrimination, Viewpoint Discrimination, 6th Amendment Right to Counsel, Interstate Commerce, Equal Rights, Due Process, Supremacy Clause. Serving the United States of America.
Website by Consultwebs.com, Inc. | Websites for Lawyers