Activating an AI-powered chatbot for staff training

Adapted from "EVA, A Chatbot for Court Staff" in AI Readiness for State Courts.

What the AI tool does

EVA is an AI-powered smart chatbot designed to retrieve accurate answers from a curated knowledge base using natural language queries. It began as a tool for civil procedures but has since expanded to other domains like criminal, probate, family law, and juvenile. EVA operates primarily on unstructured data, allowing it to provide responses from existing operational procedures. Its key features include semantic search, citation of sources, and adaptability to any knowledge base, making it a scalable and foundational tool for internal and potentially public-facing use.

Identifying the need

California's Orange County Superior Court was experiencing a growing knowledge gap due to high staff turnover and retirements, with about 50 percent of its entry-level staff being new to their role or serving less than two years. The court needed to accelerate training for new staff who were often placed in public-facing roles with limited preparation due to staffing shortages. 

Design, development & testing

The design of EVA began with a clear focus on solving a foundational business problem — closing the knowledge gap among staff — while also ensuring the solution would be scalable and user friendly. A key design principle was to make the tool accessible to non-technical users, allowing operational staff to upload documents, configure settings, and manage content without advanced programming skills. Testing was led by subject-matter experts and analyst teams from the operational units, who rigorously evaluated the tool's performance and accuracy. 

Implementation & evaluation

The rollout was incremental, starting with a small test group before expanding more broadly across departments. To support adoption, the court implemented a robust change management strategy that included training videos, internal roadshows, promotional materials like stickers, and behavioral nudges — such as encouraging staff to consult EVA before asking supervisors. Integration into Microsoft Teams made the tool easy to access, reinforcing daily use and helping to embed it into the court's workflow.

To measure success, the court tracked several key metrics: user adoption rates, the accuracy and completeness of responses, and overall utility to the end user. A simple feedback mechanism — thumbs up or down — allowed staff to rate responses, and this input was monitored by subject-matter experts who updated the knowledge base as needed. 

Monitoring, maintaining & updating

To sustain the EVA project over time, the court committed to a combination of initial development resources and ongoing operational support. The initial investment involved building the foundational technology, which was a one-time cost. Ongoing maintenance is relatively minimal from a technical standpoint and is largely absorbed into existing workflows. Most of the long-term effort involves keeping the knowledge base current — something already handled by training and analyst teams as part of their regular duties.

Because EVA is designed to be user friendly and low maintenance, it doesn't require a large dedicated team to support it. Instead, existing staff monitor usage, update procedures, and respond to feedback. The project is "resource positive," meaning it ultimately saves time and effort by reducing repetitive questions and accelerating staff training. If another organization were to purchase a similar tool today, the return on investment would likely be quick, making it a sustainable and cost-effective solution.

What's next

Since its launch, the EVA project has delivered strong outcomes and has significantly expanded in scope. Initially developed for civil procedures, EVA has since been scaled to support other domains such as criminal, probate, family law, and juvenile. Its capabilities have also evolved — such as being able to dynamically move between the different case contexts. Although it currently serves internal users, the court envisions expanding EVA to public-facing services, particularly in areas like self-help and multilingual support, where it can answer questions in languages such as Spanish and Vietnamese.

The success of EVA has also helped surface outdated procedures, prompting updates and improving consistency across departments. Its growing utility and adaptability have positioned it as a foundational tool with the potential to transform both internal operations and public service delivery.