Foreword

ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work.

The procedures used to develop this document and those intended for its further maintenance are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the different types of document should be noted. This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or www.iec.ch/members_experts/refdocs).

ISO and IEC draw attention to the possibility that the implementation of this document may involve the use of (a) patent(s). ISO and IEC take no position concerning the evidence, validity or applicability of any claimed patent rights in respect thereof. As of the date of publication of this document, ISO and IEC had not received notice of (a) patent(s) which may be required to implement this document. However, implementers are cautioned that this may not represent the latest information, which may be obtained from the patent database available at www.iso.org/patents and https://patents.iec.ch. ISO and IEC shall not be held responsible for identifying any or all such patent rights.

Any trade name used in this document is information given for the convenience of users and does not constitute an endorsement.

For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and expressions related to conformity assessment, as well as information about ISO's adherence to the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards.

This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 24, Computer graphics, image processing and environmental data representation.

Any feedback or questions on this document should be directed to the user’s national standards body. A complete listing of these bodies can be found at www.iso.org/members.html and www.iec.ch/national-committees.

Introduction

In the development of location-based service and application such as navigation, traffic line analysis, one of the most important parts is the localization and tracking method (LTM) used for estimating the position of the target people or objects. In contrast to the outdoor, global navigation satellite systems cannot be used in indoor environment because the signals from the satellite cannot be reached. In order to provide the solution for localization and tracking in indoor environment, the research and development on integrated indoor LTMs is flourishing and many new algorithms are proposed every year. Especially, dead reckoning technologies (xDR) such as pedestrian dead reckoning (PDR) can contribute integrated indoor localization, because of their capability of relative tracking without any infrastructure in the target environment.

Therefore, this document intends to foster objective evaluation and comparison of diverse LTMs, especially for targeting integrated LTMs using dead reckoning.

The target audience of this document includes stakeholders of benchmarking activities. The following are examples of how this document can be used directly or indirectly:

by a benchmarking service provider, a benchmark provider or a benchmarking competition organizer who wishes to align their benchmarking activities including self-benchmarking and open or closed competitions to be consistent with this document;
by a technology developer or supplier who wishes to estimate and evaluate the performance of an indoor localization and tracking methods appropriately with a benchmarking service provider, a benchmark provider or a benchmarking competition organizer who aligns their benchmarking activities to be consistent with this document; or
by a technology user who wishes to obtain benchmarking results based on a benchmarking activity, which is consistent with this document, or to compare the existing indoor localization and tracking methods in terms of their performance.

The main targets of the benchmarking framework are integrated indoor LTMs using dead reckoning, which are assumed to be running on the mobile devices such as smartphones.

Master sets and test environments:
The framework defines reference master sets and test environments, where integrated indoor localization methods using dead reckoning is assumed to be utilized. The reference master sets include dataset designed for evaluating individual contribution of dead reckoning method.
Benchmark metrics:
The framework defines benchmark metrics consisting of indicators for dead reckoning and other indicators evaluation practical performance of indoor localization methods.
Benchmarking process:
The framework defines benchmarking process which can evaluate each elemental performance of the contents of the integrated indoor localization methods and generalization performance.
Conformance.
The framework defines conformance check list which can clarify how each benchmarking activity conforms to the benchmarking framework.

1 Scope

This document specifies the reference framework for the benchmarking of integrated indoor localization and tracking methods (LTMs) using dead reckoning in areas of:

master sets and test environments;
benchmark metrics;
benchmarking process;
conformance.

2 Normative references

3 Terms and definitions

3.1 Terms and definitions

3.2 Acronyms and abbreviated terms

4 Overview of the framework

5 Benchmarking processes

5.1 Overview

5.2 Benchmarking flow

5.3 Stakeholders

5.4 Benchmarking types

6 Benchmark metrics

6.1 Overview

6.2 Indicators related to position

6.3 Indicators related to series of positional estimation

6.4 Indicators related to orientation

6.5 Indicators related to relative relationship

6.6 Indicator related to integrated localization

6.7 Indicators related to practicability of localization system

6.8 Indicators related to data used for the benchmarking

7 Master set for benchmarking

7.1 Overview

7.2 Dataset for off-site benchmarking

8 Conformance

Bibliography