National foreword

This British Standard is the UK implementation of ISO/IEC 19795‑10:2024.

The UK participation in its preparation was entrusted to Technical Committee IST/44/-/5, Biometric testing and reporting.

A list of organizations represented on this committee can be obtained on request to its committee manager.

Contractual and legal considerations

This publication has been prepared in good faith, however no representation, warranty, assurance or undertaking (express or implied) is or will be made, and no responsibility or liability is or will be accepted by BSI in relation to the adequacy, accuracy, completeness or reasonableness of this publication. All and any such responsibility and liability is expressly disclaimed to the full extent permitted by the law.

This publication is provided as is, and is to be used at the recipient’s own risk.

The recipient is advised to consider seeking professional guidance with respect to its use of this publication.

This publication is not intended to constitute a contract. Users are responsible for its correct application.

Compliance with a British Standard cannot confer immunity from legal obligations.

This British Standard was published under the authority of the Standards Policy and Strategy Committee on 31 October 2024.

Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.

The procedures used to develop this document and those intended for its further maintenance are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).

ISO and IEC draw attention to the possibility that the implementation of this document may involve the use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not received notice of (a) patent(s) which may be required to implement this document. However, implementers are cautioned that this may not represent the latest information, which may be obtained from the patent database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held responsible for identifying any or all such patent rights.

Any trade name used in this document is information given for the convenience of users and does not constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions related to conformity assessment, as well as information about ISO's adherence to the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 37, Biometrics.

A list of all parts in the ISO/IEC 19795 series can be found on the ISO and IEC websites.

Any feedback or questions on this document should be directed to the user’s national standards body. A complete listing of these bodies can be found at www.iso.org/members.html and www.iec.ch/national-committees.

Introduction

As the use of biometric technology increases, so too does public interest in establishing whether the technology performs similarly for all individuals. Stakeholders are asking government and industry organizations that use biometric technology to establish whether these technologies vary in performance for different demographic groups. The intention of this document is to provide guidance on how to measure and report performance variation across demographic groups.^[²^]

This document is intended to help organizations evaluate demographic performance in biometric systems and report their results. Specifically, this document outlines how to measure and report biometric performance variations across demographic groups. It provides a set of metrics and best practices to facilitate such testing. However, this document does not provide guidance on how to establish specific causes for the observed variations. The following demographic variables are explicitly discussed in this document:^[⁷^][¹⁰^][¹²^]

biological characteristics, such as:
- sex, age, weight, height and skin lightness;
social constructs, such as:
- ethnicity, gender and language.

Many other variables can cause systematic changes in biometric characteristics or in how individuals interact with biometric systems. The following demographic variables are relevant although not explicitly discussed in this document:

performance variations based on temporary states, such as:
- self-styling (e.g. makeup, eyewear, mask-wearing, clothing, hairstyles),
- behavioural or emotional states (e.g. intoxication),
- behaviours (e.g. smiling, closing eyes, varying pose);
performance variation caused by diseases or injuries, such as:
- eye surgery, cataracts, vision correction,
- stroke, cleft lip, Apert’s syndrome,
- missing digits;
performance variation caused by disabilities.

Demographic performance variation for applications other than biometric recognition, such as emotion, gender or age estimation, are not considered in this document.

1 Scope

This document establishes requirements for estimating and reporting on performance variations observed when cohorts belonging to different demographic groups engage with biometric enrolment and recognition systems. In this context, performance refers to failure-to-enrol rate, failure-to-acquire rate, shifts in comparison score, recognition error rates, and aspects of response and processing time (throughput).

This document is applicable to the following:

demographic group membership;
using phenotypic measures;
reporting on tests;
stating statistical uncertainty estimates;
operational thresholds settings;
equitability;
procurement agency activities.

This document also provides terms and definitions to be used when reporting performance variation across demographic groups.

This document is applicable to:

technology evaluations of algorithms, subsystems and systems;
scenario evaluations of systems;
operational evaluations of fielded systems.

Application of this document does not require detailed knowledge of a system’s algorithms but it does require specific knowledge of the demographic characteristics for the population of interest.

2 Normative references

3 Terms and definitions

4 Conformance

5 Planning the evaluation

5.1 Identifying the scope of the evaluation

5.2 Demographic variables

6 Executing the evaluation

6.1 Generation of mated comparison and identification trials

6.2 Generation of non-mated comparison and identification trials

6.3 Selection of a threshold

6.4 Calculating differential performance based on categorical variables for two specific demographic groups

6.5 Calculating differential performance based on categorical variables for more than two groups

6.6 Calculating differential performance in identification trials

6.7 Calculating demographic differentials for failure-to-enrol rate, failure-to-acquire rate and transaction duration

6.8 Calculating demographic differentials for continuous variables

6.9 Comparison score differential measures

6.10 Calculating uncertainty

7 Reporting the evaluation results

7.1 Reporting the experimental design

7.2 Reporting the target application

7.3 Reporting the test population

7.4 Reporting differential performance

7.5 Reporting comparison score differential measures

7.6 Reporting exception handling

Bibliography

BS ISO/IEC 19795-10:2024

Information technology. Biometric performance testing and reporting - Quantifying biometric system performance variation across demographic groups

Current, Under Review

Published:

31 Oct 2024