Evaluation of AI Solutions in Health Care Organizations — The OPTICA Tool

Abstract

Regulatory bodies are struggling to determine effective ways to regulate artificial intelligence (AI)-driven health care solutions, which repeatedly exhibit suboptimal performance and unexpected outcomes when used in new settings. Existing evaluation frameworks are typically structured as textual discussions that are difficult to translate into practical assessments of AI solutions and that often fail to consider the perspectives of specific populations and data in a designated deployment setting. Health care organizations, faced with an increasing deluge of AI solutions offerings, need a down-to-earth, executable framework that will enable setting-specific assessments of the appropriateness of AI-driven solutions. This need also arose at Clalit Health Services, a large public health care organization where AI solutions have been integrated into care for more than a decade. In response, we developed a comprehensive, practical checklist tool to assess AI solutions in health care organizations. The checklist, named OPTICA (Organizational PerspecTIve Checklist for AI solutions adoption), comprises 13 chapters, each containing 3 to 12 checklist items. We identified five main stakeholders who would generally be required to participate in the checklist completion, defined which checklist items should be completed by each stakeholder, and designated a completion order based on dependencies between checklist items. OPTICA, which has already been tested in a variety of cases, provides a practical, structured, end-to-end process for evaluating AI solutions in new clinical settings, from the unique perspective of the implementing organization.