Introduction

This document standardizes the specification of a semantic annotation scheme for visual information, based on a modelling language for constructing three-dimensional (3D) visualizations of concepts denoted by natural language (NL) expressions. This modelling language serves as a semantic basis of interpreting the semantic forms of annotation structures model-theoretically by constraining the models for interpretation. This document focuses on the introduction of the modelling language as a semantic basis for interpretation, since the syntactic specification of the annotation scheme for visual information is a simplified formulation based on the abstract specification of the spatio-temporal annotation schemes, such as those specified in ISO 24617‑1, ISO 24617‑7 and ISO 24617‑14. These three standards lay a theoretical basis for this document, which specifies ways of annotating visual information involving motions and actions that are spatio-temporally characterized.

The modelling language, named “VoxML” (visual object concept structure modelling language), where “Vox” abbreviates “visual object concept structure” (VOCS), can be used as the platform for creating multimodal semantic simulations in the context of human-computer communication. VoxML encodes semantic knowledge of real-world objects represented as 3D models, and of events and attributes related to and enacted over these objects. VoxML is intended to overcome the limitations of existing 3D visual markup languages by allowing for the encoding of a broad range of semantic knowledge that can be exploited by a variety of systems and platforms, leading to multimodal simulations of real-world scenarios using conceptual objects that represent their semantic values.

NOTE 1 The main content of this document is based on References [1] and [2]. Reference [1] was developed by the Brandeis University Computer Science Department in the context of communicating with computers (CwC), a Defence Advanced Research Projects Agency (DARPA) effort to identify and construct computational semantic elements, for the purpose of carrying out joint plans between a human and computer through NL discourse.

NOTE 2 This document adopts VoxML as a semantic basis for enriching the model for interpreting the descriptions of objects, actions and relations involving dynamic visual information.

This document outlines a specification:

a) to formulate the annotation scheme for visual information;
b) to represent semantic knowledge of real-world objects represented as 3D models.

It uses a combination of parameters that can be determined from the object’s geometrical properties as well as lexical information from NL, with methods of correlating the two where applicable. This information allows for visualization and simulation software to fill in information missing from the NL input and allows the software to render a functional visualization of programs being run over objects in a robust and extensible way. Currently, a voxicon, which is the structured repository of visual object concepts, contains 500 object (noun) voxemes, lexemes or entries of the voxicon, and 10 program (verb) voxemes.

NOTE 3 As this library of available voxemes continues to grow, the specification elements will operationalize an increasingly large library of various and more complicated programs. A voxeme library and visualization software where users will be able to conduct visualizations of available behaviours driven by VoxML after parsing and interpretation is available from Reference [25].

BS ISO 24617-10:2024

Language resource management. Semantic annotation framework (SemAF) - Visual information

Current

Published:

31 Aug 2024