Introduction

The presentation of an artefact or of human characteristics to a biometric capture subsystem in a fashion intended to interfere with system policy is referred to as a presentation attack. The ISO/IEC 30107 series deals with techniques for the automated detection of presentation attacks. These techniques are called presentation attack detection (PAD) mechanisms.

As is the case for biometric recognition, PAD mechanisms are subject to false positive and false negative errors. False positive errors wrongly categorize bona fide presentations as attack presentations, potentially flagging or inconveniencing legitimate users. False negative errors wrongly categorize presentation attacks (also known as attack presentations) as bona fide presentations, potentially resulting in a security breach.

Therefore, the decision to use a specific implementation of PAD will depend on the requirements of the application and consideration of the trade-offs with respect to security, evidence strength and efficiency.

The purpose of this document is as follows:

— to define terms related to biometric PAD testing and reporting, and
— to specify principles and methods of performance assessment of biometric PAD, including metrics.

This document is directed at vendors or test laboratories seeking to conduct evaluations of PAD mechanisms.

Biometric performance testing terminology, practices and methodologies for statistical analysis have been standardized through ISO and Common Criteria. False accept rate (FAR), false reject rate (FRR) and failure to enrol rate (FTE) are widely used to characterize biometric system performance. Biometric performance testing terminology, practices and methodologies for statistical analysis are only partially applicable to the evaluation of PAD mechanisms due to significant fundamental differences between biometric performance testing concepts and PAD mechanism testing concepts. These differences can be categorized as follows.

a) Statistical significance

Biometric performance testing utilizes a statistically significant number of test subjects, representative of the targeted user group. Error rates are not expected to vary significantly when adding more test subjects or using a completely different group.

In PAD testing, many biometric modalities can be attacked by a large or indeterminate number of potential presentation attack instrument species (PAIS). In these cases, it is very difficult or even impossible to have a comprehensive model of all possible presentation attack instruments (PAIs). Hence, it could be impossible to find a representative set of PAIS for the evaluation. Therefore, measured error rates of one set of PAIs cannot be assumed to be applicable to a different set.

PAIS present a source of systematic variation in a test. Different PAIs can have significantly different error rates. Additionally, within any given PAIS, there is random variation across instances of the PAI series. The number of presentations required for a statistically significant test scales linearly with the number of PAIS of interest. Within each PAIS, the uncertainty associated with a PAD error rate estimate depends on the number of artefacts tested and the number of individuals.

EXAMPLE 1

In fingerprint biometrics, many potent artefact materials are known, but any material or material mixture that can present fingerprint features to a biometric capture device is a possible candidate. Since artefact properties such as age, thickness, moisture, temperature, mixture rates and manufacturing practices can have a significant influence on the output of the PAD mechanism, it is easy to define tens of thousands of PAIS using current materials. Hundreds of thousands of presentations would be needed for a proper statistical analysis, and even then, resulting error rates cannot be transferred to the next set of new materials.

PAI presentation can also be source of variation in a test. Variation in pressure, position or even PAI presenter characteristics can impact PAD performance.

b) Comparability of test results across systems

In biometric performance testing, application-specific error rates based on the same corpus of biometric samples can be used to compare different biometric systems or different configurations. Results can be used to unambiguously compare and assess system performance. By contrast, when using error rates to benchmark PAD mechanisms, interpreting results can be highly dependent on the intended application.

EXAMPLE 2

In a given testing scenario with 10 PAIS (presented 100 times), System₁ detects 90 % of attack presentations and System₂ detects 85 %. System₁ detects all presentations for 9 PAIS but fails to detect all presentations with the 10th PAIS. System₂ detects 85 % of all PAIS. Which is better? In a security analysis System₁ would be worse than System₂, because revealing the 10th PAIS would orient an attacker such that they could use this method to defeat the capture device all the time. However, if attackers could be prevented from using the 10th PAIS, System₁ would be better than System₂, because individual rates indicate that it is possible to overcome System₂ with all PAIS.

c) Cooperation

Many biometric performance tests address applications such as access control in which subjects are cooperative. Errors due to incorrect operation are an issue of a lack of knowledge, experience or guidance rather than intent. Significant uncooperative behaviour in a group is not part of the underlying “biometric model” and would render the determined error rates almost useless for biometric performance testing.

PAD tests include subjects whose behaviour is not cooperative. Attackers will try to find and exploit any weakness of the biometric system, circumventing or manipulating its intended operation. Presentation attack types, based on the experience and knowledge of the tester, can change the success rates for an attack dramatically. Hence it can be difficult to define testing procedures that measure error rates in a fashion representative of cooperative behaviour.

d) Automated testing

In biometric performance testing, it is often possible to test comparison algorithms using databases from devices or sensors of similar quality. Performance can be measured in a technology evaluation using previously collected corpora of samples as specified in ISO/IEC 19795‑1.

In PAD testing, data from the biometric capture device (e.g. digitized fingerprint images) can in some cases be insufficient to conduct evaluations. Biometric systems with PAD mechanisms often contain additional sensors to detect specific properties of a biometric characteristic. Hence, a database previously collected for a specific biometric system or configuration is not necessarily suitable for another biometric system or configuration.

Even slight changes in the hardware or software could make earlier measurements useless. It is generally impractical to store multivariate synchronized PAD signals and replay them in automated testing. Therefore, automated testing is often not an option for testing and evaluating PAD mechanisms.

e) Quality and performance

In biometric performance testing, performance is usually linked directly to biometric data quality. Low-quality samples generally result in higher error rates while a test with only high-quality samples will generally result in lower error rates. Quality metrics are therefore often used to improve performance (dependent on the application).

In PAD testing, even though low biometric quality can cause an artefact to be unsuccessful, there is no reason to assume a certain quality level from artefacts in general. Samples from artefacts can exhibit better quality than samples from human biometric characteristics. Without a model of attacker skill, it seems valid (at least in a security evaluation) to assume a “worst case” scenario where the attacker always uses the best possible quality. That way, one can at least determine a guaranteed minimal detection rate for the specific test set while reducing the number of necessary tests at the same time. It is then a matter of rating the attack potential of successful artefacts (effort and expertise for the needed quality) in order to assess the security level, as is the practice in Common Criteria evaluations.

Based on the differences in a) through e), the following general comments regarding error rates and metrics related to PAD mechanisms can be derived.

— In an evaluation, PAIS are analysed/rated separately.
— Attack presentation classification error rates other than 0 % for a PAIS only prove that the PAI can be successful. A different tester can potentially achieve a higher or lower attack presentation classification error rate. Further, training to identify the relevant material and presentation parameters could increase the attack presentation classification error rate for this PAIS. The experience and knowledge of the tester, as well as the availability of the necessary resources, are significant factors in PAD testing and are taken into account when conducting comparisons or performance analysis.

Error rates for PAD mechanisms are determined by the specific context of the given PAD mechanism, the set of PAIS, the application, the test approach, and the tester. Error rates for PAD mechanisms are not necessarily comparable across similar tests, and error rates for PAD mechanisms are not necessarily reproducible by different test laboratories.

BS ISO/IEC 30107-3:2023

Information technology. Biometric presentation attack detection - Testing and reporting

Current

Published:

31 Jan 2023