Introduction

Artificial Intelligence (AI) can be defined as the capability of a system to perform tasks or develop data processing systems that perform functions normally associated with human intelligence (adapted from ISO/IEC 2382:2015).

AI can bring benefits to healthcare, including improved clinical outcomes, improved efficiencies and improvement in the management of healthcare itself. However, the implementation of new technologies, such as AI, can also present risks that could jeopardize patient health and safety, increase inequalities and inefficiencies, undermine trust in healthcare, and adversely impact the management of healthcare.

A 2020 joint AAMI/BSI whitepaper [1] explored the safety and effectiveness of AI in medical devices and concluded that data-driven AI medical device systems or machine learning (ML) medical device systems differed from traditional rules-based systems in three ways:

1) Learning — these systems can accumulate data to provide an elaboration that could have a positive impact on patient health in any term defined by the intended purpose of the medical device (in vitro or not) according to relevant regulations;
2) Autonomy — these systems have the potential to modify processes and outputs in response to their learning with reduced or even without clinician oversight; and
3) Inexplicability1) — because of their sophisticated computational abilities, the intricate statistics involved, and the large, complex datasets involved, the rationale for outputs produced by such systems might not be easily understood by well-trained clinicians and other healthcare personnel, let alone by individuals without specialist knowledge [1].

Despite the sophistication and complicated methodologies employed, ML systems can introduce risks to safety by learning incorrectly, making wrong inferences, and then recommending or initiating actions that can lead to harm. The amplification of errors in an AI system has the potential to create large-scale harm to patients. Sometimes these systems detect correlations in data sets instead of causations, which can lead to incorrect conclusions. One well-known example of correlation identification is an image recognition system that was trained to differentiate between photos of a wolf compared to photos of a husky dog. The wolf images used to train the system often had snow in the background, and the software picked up on this correlation, instead of detecting the differences between the animals themselves [2].

All medical devices come with inherent risks. Manufacturers are required to demonstrate that their medical devices do not pose unacceptable risks, and that the benefits of their intended use outweigh the overall residual risk. BS EN ISO 14971 details how manufacturers assess and mitigate potential risks in order to protect the health and safety of patients as well as data and system security. Additionally, BS EN IEC 80001‑1 addresses risks for IT networks incorporating medical devices and PD IEC/TR 80002‑1 addresses software internal to the medical device that might support AI. These standards, and associated supporting guidance documents, provide the basis for risk management and the life cycle process for all software that is regulated under medical device legislation.

To manage a risk, however, one has to be aware that the risk exists. Risks are often hypothesized from clinical knowledge of the anatomy, physiology or condition and real-world experience with similar medical devices. With AI-enabled medical devices, real-world experience with similar technologies might be lacking, increasing the potential to fail to identify hazards, and inequities, misunderstand the technology or the process or underestimate the risk.

One of the key findings of the 2020 AAMI/BSI whitepaper [1] was the need to develop “risk management guidance to assist in applying_BS EN ISO 14971 to AI as a medical technology”. BS EN ISO 14971 provides a process for managing the risk associated with medical devices. It has been recognized by medical device regulators and adopted as a national standard in countries across the world. This British Standard/AAMI Technical Information Report does not provide a new risk management process, nor does it expand the requirements of BS EN ISO 14971. Rather, it provides guidance to assist those who are applying BS EN ISO 14971 to regulated AI medical technologies.

There are many different algorithms that can support AI, including technologies such as decision trees, genetic algorithms and neural nets. Often, when people are discussing AI, they are specifically talking about ML systems. Since ML systems are often more complicated and more opaque than other approaches, this document focuses on ML-related risks. Because of the potential for confusion, the remainder of this document avoids the use of the terms “AI” and “AI-enabled” and uses ML.

BS/AAMI 34971:2023

Application of ISO 14971 to machine learning in artificial intelligence. Guide

Current

Published:

31 May 2023