Foreword

Publishing information

This joint AAMI and British Standard is published by BSI Standards Limited, under licence from The British Standards Institution, and came into effect on 31 May 2023. It was prepared jointly by Subcommittee CH/210/4, Risk analysis for Medical Devices, under the authority of Technical Committee CH/210, Quality management and corresponding general aspects for medical devices, and the AAMI Artificial Intelligence Committee. A list of organizations represented on the BSI committees can be obtained on request to the committee manager.

Relationship with other publications

This publication provides guidance for applying BS EN ISO 14971.

Information about this document

This publication was developed jointly with the Association for the Advancement of Medical Instrumentation (AAMI) as part of the AAMI/BSI initiative on artificial intelligence (AI).

Copyright is held jointly between the British Standards Institution and AAMI.

This publication can be withdrawn, revised, partially superseded or superseded. Information regarding the status of this publication can be found in the Standards Catalogue on the BSI website at bsigroup.com/standards, or by contacting the Customer Services team.

Where websites and webpages have been cited, they are provided for ease of reference and are correct at the time of publication. The location of a webpage or website, or its contents, cannot be guaranteed.

Use of this document

As a guide, this British Standard takes the form of guidance and recommendations. It should not be quoted as if it were a specification or a code of practice.

Presentational conventions

The guidance in this standard is presented in roman (i.e. upright) type. Any recommendations are expressed in sentences in which the principal auxiliary verb is “should”.

Commentary, explanation and general informative material is presented in smaller italic type, and does not constitute a normative element.

Where words have alternative spellings, the preferred spelling of the Shorter Oxford English Dictionary is used (e.g. “organization” rather than “organisation”).

Contractual and legal considerations

This publication has been prepared in good faith, however no representation, warranty, assurance or undertaking (express or implied) is or will be made, and no responsibility or liability is or will be accepted by BSI in relation to the adequacy, accuracy, completeness or reasonableness of this publication. All and any such responsibility and liability is expressly disclaimed to the full extent permitted by the law.

This publication is provided as is, and is to be used at the recipient’s own risk.

The recipient is advised to consider seeking professional guidance with respect to its use of this publication.

This publication is not intended to constitute a contract. Users are responsible for its correct application.

Compliance with a British Standard cannot confer immunity from legal obligations.

AAMI Foreword

This technical information report/BSI Brtish Standard was developed based on the Consensus Report which was proposed to be written after the success of a white paper collaborated on between the Association for the Advancement of Medical Instrumentation (AAMI) and the British Standards Institution (BSI). This is being developed to provide guidance on machine learning as it applies to ISO 14971:2019.

This document was first developed as an AAMI Consensus Report (CR) but has also been provided to BSI for comments and input. BSI and AAMI then developed this as a BSI British Standard and AAMI Technical Information Report (TIR).

As part of the consensus-driven process of the AAMI standards program, this document was circulated for comments to the AAMI Artificial Intelligence Committee and the BSI CH/210/4 Risk Analysis for Medical Devices. This document was also balloted for approval by the AAMI Artificial Intelligence Committee.

Suggestions for improving this Technical Information Report are invited. Comments and suggested revisions should be sent to Standards, AAMI, 900 North Glebe Road, Suite 300, Arlington, VA 22203-1633.

NOTE This foreword does not contain provisions of the AAMI TIR34971, Application of ISO 14971 to machine learning in artificial intelligence – Guide (AAMI TIR34971:2023), but it does provide important information about the development and intended use of the document.

Introduction

Artificial Intelligence (AI) can be defined as the capability of a system to perform tasks or develop data processing systems that perform functions normally associated with human intelligence (adapted from ISO/IEC 2382:2015).

AI can bring benefits to healthcare, including improved clinical outcomes, improved efficiencies and improvement in the management of healthcare itself. However, the implementation of new technologies, such as AI, can also present risks that could jeopardize patient health and safety, increase inequalities and inefficiencies, undermine trust in healthcare, and adversely impact the management of healthcare.

A 2020 joint AAMI/BSI whitepaper [1] explored the safety and effectiveness of AI in medical devices and concluded that data-driven AI medical device systems or machine learning (ML) medical device systems differed from traditional rules-based systems in three ways:

Learning — these systems can accumulate data to provide an elaboration that could have a positive impact on patient health in any term defined by the intended purpose of the medical device (in vitro or not) according to relevant regulations;
Autonomy — these systems have the potential to modify processes and outputs in response to their learning with reduced or even without clinician oversight; and
Inexplicability1) — because of their sophisticated computational abilities, the intricate statistics involved, and the large, complex datasets involved, the rationale for outputs produced by such systems might not be easily understood by well-trained clinicians and other healthcare personnel, let alone by individuals without specialist knowledge [1].

Despite the sophistication and complicated methodologies employed, ML systems can introduce risks to safety by learning incorrectly, making wrong inferences, and then recommending or initiating actions that can lead to harm. The amplification of errors in an AI system has the potential to create large-scale harm to patients. Sometimes these systems detect correlations in data sets instead of causations, which can lead to incorrect conclusions. One well-known example of correlation identification is an image recognition system that was trained to differentiate between photos of a wolf compared to photos of a husky dog. The wolf images used to train the system often had snow in the background, and the software picked up on this correlation, instead of detecting the differences between the animals themselves [2].

All medical devices come with inherent risks. Manufacturers are required to demonstrate that their medical devices do not pose unacceptable risks, and that the benefits of their intended use outweigh the overall residual risk. BS EN ISO 14971 details how manufacturers assess and mitigate potential risks in order to protect the health and safety of patients as well as data and system security. Additionally, BS EN IEC 80001‑1 addresses risks for IT networks incorporating medical devices and PD IEC/TR 80002‑1 addresses software internal to the medical device that might support AI. These standards, and associated supporting guidance documents, provide the basis for risk management and the life cycle process for all software that is regulated under medical device legislation.

To manage a risk, however, one has to be aware that the risk exists. Risks are often hypothesized from clinical knowledge of the anatomy, physiology or condition and real-world experience with similar medical devices. With AI-enabled medical devices, real-world experience with similar technologies might be lacking, increasing the potential to fail to identify hazards, and inequities, misunderstand the technology or the process or underestimate the risk.

One of the key findings of the 2020 AAMI/BSI whitepaper [1] was the need to develop “risk management guidance to assist in applying_BS EN ISO 14971 to AI as a medical technology”. BS EN ISO 14971 provides a process for managing the risk associated with medical devices. It has been recognized by medical device regulators and adopted as a national standard in countries across the world. This British Standard/AAMI Technical Information Report does not provide a new risk management process, nor does it expand the requirements of BS EN ISO 14971. Rather, it provides guidance to assist those who are applying BS EN ISO 14971 to regulated AI medical technologies.

There are many different algorithms that can support AI, including technologies such as decision trees, genetic algorithms and neural nets. Often, when people are discussing AI, they are specifically talking about ML systems. Since ML systems are often more complicated and more opaque than other approaches, this document focuses on ML-related risks. Because of the potential for confusion, the remainder of this document avoids the use of the terms “AI” and “AI-enabled” and uses ML.

1 Scope

This joint AAMI and British Standard provides guidance for applying a BS EN ISO 14971 risk management process when evaluating medical technology utilizing machine learning (ML). It is intended to apply to ML-enabled medical devices2) throughout all phases of the product life cycle.

This document is intended to be used in conjunction with BS EN ISO 14971. It does not modify the BS EN ISO 14971 risk management process; rather it provides information and guidance to inform the application of BS EN ISO 14971 to ML medical technology. A risk management process is further detailed in Annex A.

This British Standard addresses the same types of risk that are addressed in BS EN ISO 14971 but focuses on risks that are elevated with or unique to ML medical devices. Because artificial intelligence (AI) and ML are software-driven, the unique or elevated risks are those around data management, feature extraction, algorithm training, evaluation, bias, health inequity, safety, and cyber and information security. This standard also provides examples and suggests strategies for eliminating or mitigating the associated risk.

Annex B provides additional clarification and examples of hazards, hazardous situations and harms, as well as possible risk control strategies in a series of tables that correspond with BS EN ISO 14971:2019+A11, Table C.1, Table C.2, and Table C.3.

2 Normative references

3 Terms and definitions

4 General requirements for risk management system

4.1 Risk management process

4.2 Management responsibilities

4.3 Competence of personnel

4.4 Risk management plan

4.5 Risk management file

5 Risk analysis

5.1 Risk analysis process

5.2 Intended use and reasonably foreseeable misuse

5.3 Identification of characteristics related to safety

5.4 Identification of hazards and hazardous situations

5.5 Risk estimation

6 Risk evaluation

7 Risk control

7.1

7.2

7.3

8 Evaluation of overall residual risk

8.1

8.2

8.3

8.4

8.5

9 Risk management review

9.1

9.2

10 Production and post-production activities

10.1

10.2

10.3

Bibliography

BS/AAMI 34971:2023

Application of ISO 14971 to machine learning in artificial intelligence. Guide

Current

Published:

31 May 2023