Investments in Human Capital Pay Dividends for Courts

Every year, state judicial systems face an increasingly diverse population grappling with a rapidly expanding and ever-evolving body of law.  Our system races to incorporate new technology to provide the instant service the public increasingly expects while state budgets continue to decrease.

M. Christy Tull
Manager, Curriculum Development, Supreme Court of Ohio Judicial College

Ann A. O’Connell
Professor, Quantitative Research, Evaluation, and Measurement, School of Educational Policy and Leadership, College of Education and Human Ecology, The Ohio State University

Among the effects of our recession-starved budgets is the growing need to equip and educate new judges and judicial staff, as well as those judges and staff overburdened by changing roles or larger workloads. Staffing cuts are taking their toll on both judges and staff. With 34 states having laid off court employees, and 39 states not filling clerk vacancies (Schwartz, 2011), the remaining staff, who must do more with less, need cross-training to maintain services competently. In addition, a growing number of seasoned judges and staff are retiring earlier to secure existing benefits (OPERS, 2011). However, judicial vacancies in 26 states are going unfilled (USA Today, 2011); in some states, the option for help from retired judges sitting by assignment has been eliminated, adding to the strain on judges’ workload (Thomson Reuters, 2012). This downsizing, together with unrepresented litigants, new sciences, changing technologies, and increasing demands on courts, results in overburdened and underequipped judges and staff who strive to carry out the administration of justice efficiently and effectively.

In the best or worst of times, judicial branch education exists to meet the demand of continuously updating knowledge and skills. Specifically, it aims to close or narrow the distance between existing and desired knowledge and skills, so that judges and court personnel are well equipped in their roles to serve the public interest efficiently and competently. The primary means of equipping judges and staff to meet the demands of our times is through effective judicial branch education.

This article builds on the premise that judicial branch education—when based on proven education principles—plays a critical role in the effective administration of justice. Evidence to support this premise is identified through recent evaluation findings regarding the impact of judicial branch education. A model to maximize impact of education is also provided. Finally, a sampling of practical evaluation strategies are introduced for use by court leaders or justice partners at the local, state, or national level.

Purpose of Educating the Court’s Human Capital

A 1996 quote by Livingston Armytage neatly summarizes the goal for judicial education: “The purpose of any program of continuing judicial education is to provide a process . . . to improve judicial performance, and thereby, the quality of justice” (p. 7). A similar focus on the importance of the education process for nonjudicial court staff can be found in the National Association for Court Management’s Core Competency Curriculum Guidelines, the seminal document that guides education for court managers. It states that these guidelines “recognize that neither court systems nor their constituent courts can operate efficiently or effectively without competent court leaders, professionals who understand that their and their staff’s continuing personal and professional development is a necessity, not a luxury. Personal and professional development of court leaders and their successors and staff is an investment that pays dividends year after year” (emphasis added).

Judicial Branch Education Is an “Evidence-Based” Practice

So, what evidence exists for the expected positive outcomes of judicial branch education? How do we know that investment in our courts’ human capital through judicial branch education “pays dividends year after year?”

High importance and expectations are often placed on education’s role to help achieve the court’s many purposes (e.g., serve/protect public, guard individual freedoms, interpret and apply law, provide justice, resolve disputes, check/balance). But, unlike in the private sector where educators may measure quality through increased production of widgets or profit margins, judicial branch educators face a much harder Investments in Human Capitaltask in measuring their impact on the improved administration of justice (Conner, 2002). This challenge of measuring a societal outcome is particularly affected by limited funds for conducting evaluations, together with the growing need for judicial branch education to adapt to increasing or changing workloads of judges and their staff.

Despite the challenges, however, there is a growing body of evidence that judicial branch education offers exceptional dividends in providing needed knowledge and skills. When these positive outcomes can be linked to critical elements of education theory and adult-education principles, support emerges for judicial branch education as an evidence-based practice that enhances the administration of justice.

As a recent example of the impact of judicial education, a 2011 evaluation of the Supreme Court of Ohio Judicial College’s course offerings found that participation in their courses was effective at contributing to changes in knowledge, attitudes, and skills/behaviors of judges and magistrates. This yearlong study focused specifically on professional development for juvenile court judges and magistrates. However, the findings are widely applicable to other jurisdictions and court personnel. The evaluation also found that judges form a strong community of practice through their participation in judicial branch education. That is, judges are clearly invested in their own and their peers’ continued growth and development as judges. They invest in the knowledge bank of their community—by exchanging knowledge and experience—and they draw down dividends—by learning from peers whom they view as the most credible providers of knowledge regarding their roles as judges. Again, the benefit of a community of practice extends to all adult learners, judges and court personnel alike.

Specific Findings

In early 2010, the Supreme Court of Ohio Judicial College embarked on a principled approach to understand the impact of their existing judicial branch education courses, information that is critical for identifying promising professional development strategies and improving education programming. Consequently, a partnership was established with the Ohio State University (OSU) to evaluate judge and magistrate education courses; funding to support the evaluation was obtained from the U.S. Department of Health and Human Services’ Court Improvement Program.

The specific aim of this one-year evaluation project was to investigate the impact on juvenile judges and magistrates in Ohio through their participation in the Judicial College’s abuse, neglect, and dependency continuing-education courses. The study sought to determine changes in participants’ knowledge, attitudes, and behavior/skills. The evaluation focused on existing processes and education-delivery structures and was designed to result in recommendations for improvements in course evaluation methods, instructional practice, and educational outcomes for judges and magistrates.

Researchers relied on empirical data as well as participant reflection and self-reporting to investigate the impact of judicial branch education courses on judge knowledge, attitudes, and behavior/skills. The methodology drew on Schrader and Lawless (2004) and Alexander (2003) to provide definitions of three essential aspects of learning and the development of competence. Those areas of impact and OSU’s findings follow:

  1. Knowledge : Knowledge refers to information that was acquired or actually learned as a consequence of the professional development. Knowledge is often expressed in three forms. Declarative knowledge refers to content; procedural knowledge refers to knowing how to do something; and conditional knowledge refers to knowing when or why to do something. OSU Finding: Judicial branch education courses are successful in providing new knowledge and information not previously known by learners. Examples of topical areas in which knowledge change was observed include greater knowledge of case law, legislation, and consequences of child trauma.
  2. Attitudes : Attitudes can refer to personal and often subjective dispositions, an internal philosophy or belief, and a state of readiness that can then lead to a specific behavior. Thus, attitudes represent a predisposition toward action. OSU Finding: Judicial branch education courses are effective in changing the dispositions, attitudes, philosophy, or beliefs of attendees, supporting the likelihood of actual changes in behavior. Examples of evidence for attitude change based on participation in judicial branch education courses include strong inclination to translate new content into judicial practice (i.e., predisposition to action) and an awareness of potential unfair treatment of marginalized groups (e.g., transgender persons, or lesbian/gay/bisexual persons).
  3. Behavior/Skills : Behaviors and skills represent the integration of knowledge and attitudes into actual practice. Behaviors and skills are externally observable, as contrasted with attitudes or beliefs, which are not directly observable. OSU Finding: Judicial branch education courses are effective in developing behavior/skills of juvenile judges through integration and incorporation of new learning into actual judicial practice. Multiple self-reported examples of behavior and skills change or application of new skills include improved case disposition timing, more frequent use of bench cards obtained at courses, and improved on-the-job behavior and skills based on mock-trial experiences gained from new judges’ orientation.

Finally, there was an important—and unexpected—area in which judicial branch education positively makes a difference. Professional development education helps form a “community of practice” for judges, which serves as a platform for enhancing knowledge, attitudes, and behavior change. As a result, the community of practice (e.g., networking, forum to discuss common goals or obstacles), supported by and generated through judicial branch education, acts as a catalyst for development of expertise among judges and court personnel.

Other Research Measuring Judicial Branch Education’s Impact

Evidence to support the belief that judicial branch education—when based on adult-learning principles—makes a difference is not limited to Ohio’s experiences. Investments in Human CapitalIn response to recent calls by the National Association of State Judicial Educators (NASJE) for more rigorous evaluations of judicial branch education, there have been efforts—especially on the national level—to move beyond end-of-course satisfaction surveys. A comprehensive review of the literature on judicial branch education revealed that evaluations focused on measuring impact—improvements in knowledge and consequent changes in behavior and attitudes—are more likely when the objectives for judicial branch education are linked to a stated court system mission or objectives. For example, a built-in evaluation component, tied to national objectives, was present in national evaluations on domestic violence, hard-core drunk drivers, and court management trainings. Similarly, a more advanced evaluation is often more likely at the state level when there is a state-mandated judicial performance evaluation program.

In the absence of national, state, or local directives for evaluation, however, assessments of judicial branch education gravitate more toward satisfaction evaluations (e.g., rating content, faculty, and facility), rather than toward impact evaluations. A particular challenge to providers and consumers of judicial branch education, then, is to identify practices that any provider can follow to increase the likelihood of a positive impact. In times of economic stress, or when workloads are intense or intensely changing, such a model can help judicial branch educators provide ongoing and effective evidence-based programming.

A Model to Maximize Impact

To ensure long-term, high-quality, and effective continuing judicial branch education, the following is a model to maximize impact. This model is based on the literature and empirical evaluation work and is meant to be a useful adjunct to existing planning and implementation guides for those involved in designing education for judges and court personnel (e.g., NASJE’s instructional design template). The model includes the following strategies.

  • Align desired courses and curricula with local, state, or national objectives and standards (e.g., court mission statements and strategic plans, Trial Court Performance Standards, CourTools, NACM curriculum);
  • Customize and tailor training and assessment/evaluation to individual judges’ and court personnel’s unique jurisdictional demographics and other relevant factors, such as number of years of experience;
  • Design and Deliver professional development courses and sessions to incorporate a variety of teaching techniques (e.g., avoid reliance on lecture), adult-learning principles (e.g., relevant, draws out participant experience, immediately applicable), instructional design principles (e.g., linking needs assessment, stated goals and objectives with content and delivery methods), and opportunities for engaging in and developing a community of practice;
  • Assess courses using course-specific and objectives-based methods and assess transfer of learning from course to job; and
  • Establish long-term follow-up efforts to measure effectiveness of professional development and establish real-time corrections for optimal professional development outcomes.

Investments in Human Capital

What You Can Do on a Shoestring or with a Cadillac

If you are a court leader or justice partner who recognizes the benefits of judicial branch education, the findings identified in this article are most likely of interest to you, even if you are not directly involved in education or the evaluation of it. Quality education provides a vital role by equipping judges and staff with the education and training to help them efficiently and effectively carry out the administration of justice. Evaluation is an important tool to ensure that quality education is provided and that judicial branch education contributes to accomplishing the purposes of the court.

There is a range of simple to sophisticated ways to evaluate judicial branch education. Evaluation methods can range from participant reports of their satisfaction on the day of the course, to faculty evaluating participant learning during a course, to a supervisor assessing the degree to which an individual employee changes because of education, to evaluation, at its highest level, that measures the degree of impact on the organization and society.

Most judicial branch education providers already obtain end-of-the-day course-evaluation feedback (i.e., participant satisfaction). While this information is important, especially if tied to stated objectives, other more rigorous methods of evaluation can go beyond simply measuring satisfaction. Not surprisingly, time and resources can increase accordingly with the higher level or efforts of evaluation (especially, transfer of learning and impact studies).

Deciding which type of evaluation is desirable and feasible for your court depends on the answers to the following why, what, and how questions (National Association of State Judicial Educators, 2012).

Why? No matter at what level a court or organization decides to evaluate, clearly articulated reasons for doing an evaluation are needed at the outset. The following are five compelling reasons: 1) to demonstrate the degree of success of a course or program; 2) to justify continued or enhanced funding; 3) to determine whether a course or program should continue, be changed, or improved; 4) to answer cause-and-effect questions; and 5) to focus on whether and what value is derived and what difference the course or program makes. Evaluation of any kind can raise awareness of the importance of judicial branch education to help decision makers have a greater understanding of education and how it makes a difference.

What? Decisions need to be made to answer the following questions. What is important to this court or organization to evaluate (e.g., impact of one course, one jurisdiction, one audience, or overall impact)? What can be measured? What is most beneficial to be measured? Answers to these questions and more will help determine the scope of the project and the entity evaluated.

How? The kind of evaluation that can be done goes from simple to robust, narrow to broad, short-term or long-term, qualitative to quantitative data, local to state to national-based programming. A few examples of evaluation efforts include:

  • Enhancements to existing “satisfaction” surveys can be quick fixes with rich returns. The following are a few questions that can be asked at the end of a course day or months later using an electronic survey tool:

o What have you learned in this session that you did not know previously? What barriers do you perceive,
if any, to implementing the information from this session?

o In what way, if any, do you foresee that today’s experience will affect your performance or thinking
as a judge/magistrate/court personnel?

  • Use of pre-tests and post-tests, while ensuring anonymity, can reveal some tangible findings and often can be used in real-time to make midstream corrections. This is a common tool used by faculty to assess learning and make adjustments. For example, if faculty have taught content and the learners get an answer wrong, faculty can attempt to reteach or correct understanding.
  • Incorporating multiple methods of data collection when evaluating courses or programs can yield support for change due to judicial branch education. For example, Virginia did an in-house evaluation of the impact of a “Calendar Management and Delay Reduction” education program for judges and clerks. They ensured that the course objectives were aligned specifically with the court system’s two-year strategic plan and did a pre- and post-sampling of various types of cases to determine the average length of time to final case disposition. Virginia also conducted exit surveys and face-to-face interviews (Connor, 2002).
  • State statistical data and trends can be used to demonstrate the impact of judicial branch education. Changes over time in court statistical data can sometimes be correlated with the increase in or mandate of education, as was the case in Florida. A reduction in reversal rates was linked, in part, to the new Supreme Court of Florida requirement that trial judges complete a comprehensive course on death-penalty cases, according to National Judicial College documentation.
  • Documenting spontaneous testimonials (via letters or course evaluation comments) and suggestions/information learned from comments in a focus group are other means that provide useful qualitative data.

Investments in Human CapitalObviously, the scope of an evaluation project, beyond satisfaction surveys, depends on available time and resources (i.e., people and money). A broadly based impact evaluation takes time. Ohio’s experience was relatively modest, but about two years were devoted to this project. This includes searching for an outside evaluator, scoping the project, finding funding, gaining administrative support, assisting with the data collection, and keeping all parties and stakeholders informed about the evaluation. Full implementation of the recommendations is still underway.

Before embarking on an evaluation of judicial branch education, decisions must be made to determine the pros/cons and the feasibility of an in-house evaluation team versus an external consultant. Regarding the latter, it is ideal if you can find an area university department with whom to partner. Many large universities have “service learning” offices and education, public-policy, and criminal-justice departments that might find this kind of project to be a good fit. Grant funds, such as the U.S. Department of Health and Human Services’ Court Improvement Program that Ohio used, can both narrow the scope and, in Ohio’s case, pay for a graduate student to help conduct the research.

Closing: The Bank of Education and Training

The 2009 image of a plane floating in the Hudson River with passengers standing on the wings awaiting rescue is now imprinted on Americans’ minds.

How was it possible that the plane landed and all were safe? When asked, Captain Chesley "Sully" Sullenberger said, "[F]or 42 years I've been making small regular deposits in this bank of experience: education and training. And on January 15, the balance was sufficient so that I could make a very large withdrawal." When asked about her quick reaction to this very same disaster, one ferry boat operator said “Training, training, and training.”

While a judge does not land planes or drive ferries, judges do make life-and-death decisions. Court managers and other staff affect the lives of individuals and their families who navigate through the court daily. Judicial branch education helps equip judges and court personnel in making the needed deposits into their banks of knowledge and skills so that they can make withdrawals to fulfill their roles. As they continuously improve their competence, the administration of justice is enhanced and public trust and confidence in the judicial system are upheld.