Disclaimer

This document is the copyrighted property of ASAM e.V. In alteration to the regular license terms, ASAM allows unrestricted distribution of this standard. §2 (1) of ASAM’s regular license terms is therefore substituted by the following clause: "The licensor grants everyone a basic, non-exclusive and unlimited license to use the standard ASAM OpenODD".

1. Foreword

The Association for Standardization of Automation and Measuring Systems (ASAM) is a non-profit organization that promotes standardization for tool chains in automotive development and testing. Our members are international car manufacturers, suppliers, tool vendors, engineering service providers and research institutes. ASAM standards are developed in work groups, composed of experts from our member companies. Our standards enable easy exchange of data or tools within a tool chain. ASAM is the legal owner of these standards and responsible for their development, maintenance, distribution and marketing.

ASAM standards focus on defining data models, file formats, communication APIs, software component APIs, and communication protocols for the data exchange between research, development, and validation systems. ASAM standardization seeks to incorporate requirements from many different global viewpoints and produce an efficient interface.

2. Introduction

2.1. Overview

Safety is fundamental to the development of public trust and acceptance of Connected and Autonomous Vehicles (CAVs) and their on board Automated Driving Systems (ADSs) and to enable deployment of automated driving. Safety of CAVs has two aspects: safe design and safe use of the system. In order to ensure safe use of the system, it is important to convey the knowledge of the true capabilities and limitations of the ADSs to the users to prevent misuse of the systems.

In order to establish the true capabilities and limitations of an Automated Driving Systems (ADSs), we need to first define their Operational Design Domain. An ODD refers to the operating environment (road type, weather conditions, traffic conditions) in which a vehicle can drive safely. For example, for Low-Speed Automated Driving (LSAD) systems such as pods and shuttles, the ODD may include urban areas with predefined routes that include pedestrians and cyclists. On the other hand, for a motorway chauffeur system, an ODD may include a four-lane divided motorway and dry conditions only. The types of scenarios a vehicle may encounter will be a function of its defined ODD, making this fundamental to any safety evaluation and scenario identification.

- A more formal definition of ODD as defined by SAE J3016 (2018) states that "Operating conditions under which a given driving automation system or feature thereof is specifically designed to function, including, but not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics". -

In order to enable stakeholders to share, compare and re-use ODD definitions, there is a need for standards to provide guidance to the stakeholders on both the attributes to be used for ODD definition and a format for defining the ODD using those attributes. BSI PAS 1883 (UK) provides a taxonomy for ODD. Additionally, ISO 34503 uses the taxonomy to provide a high-level definition format for ODD. While these standardization activities address the important needs of the industry, a gap still exists in the industry for an ODD definition format for simulation.

ASAM OpenODD is a representation of the abstract ODD specification in a more well-defined syntax and semantics which enables machines to interpret and perform the required analysis. Additionally, the ASAM OpenODD specification shall be measurable and verifiable for the attributes it specifies.

2.1.1. Motivation

An Operation Design Domain definition (ODD) shall be valid throughout the whole vehicle’s lifetime. The definition of an ODD is part of the safety concept of a vehicle. Depending on the current development step different information needs to be derived from such an overall ODD.

ASAM OpenODD focuses on a machine-readable format. The ODD must be represented so it can easily be used within simulation and other machine processed environments. The content of ASAM OpenODD will be derived from any abstract ODD, providing the information in a usable manner. For the purpose of using this ODD description for simulations and post-processing the format must fulfill the following requirements:

  • Searchable

  • Exchangeable

  • Extensible

  • Machine readable

  • Measurable and verifiable

  • Human readable

In the scenario based testing workflow ASAM OpenODD will play a very important role supporting the test description, defining the boundaries of what to test and achieving a good test coverage of the operational design domain and its borders.

An example view on a scenario base testing workflow

overview

For more details on scenario based testing workflows, consult the Paper released by the ASAM Test Specification Study Group.

Using the ASAM OpenODD specification will ensure that it is possible to statistically quantify:

  1. exposure of certain attribute values of the ODD,

  2. the performance of an CAV and its systems against the ODD,

  3. the tightening or expansion of the ODD over time.

ASAM OpenODD will be compatible with the wider ASAM OpenX suite of standards (OpenDRIVE, OpenSCENARIO and OpenXOntology). Furthermore, ASAM OpenODD will be consistent with the ongoing ISO activities (e.g. AWI ISO 34503). Compatibility of ASAM OpenODD will be achieved during a harmonization phase before the beggining of the standardization project.

2.2. Normative and non-normative statements and deliverables

This specification uses a standard information structure. The following rules apply regarding normativity of sections:

  • Statements expressed as requirements, permissions, or prohibitions according to the use of modal verbs, as defined in Section Modal verbs, are normative.

  • UML diagrams from the ASAM OpenODD UML model are normative.

  • Rules for ASAM OpenODD data structures in "Rules" sections are normative.

  • XML examples and use case descriptions are non-normative.

2.3. Conventions

2.3.1. Units

Unless stated otherwise, all numeric values within this specification are in SI units, for example:

  • position/distance in [m]

  • angles in [rad]

  • time in [s]

  • speed in [m/s]

Geographic positions are stated in the unit defined by the spatial coordinate system, for example, in accordance with WGS 84 – EPSG 4326 [1].

Some data elements allow to explicitly state the unit of a given value. If the unit is not given or if there is no means to state the unit, SI units apply. The following units may be used in explicit assignments:

Table 1. Units
Category Description Identifier

distance

meter

m

kilometer

km

feet

ft

land mile

mile

speed

meters per second

m/s

miles per hour

mph

kilometers per hour

km/h

mass

kilogram

kg

metric tons

t

slope

percent

%

These optional units shall be used for purposes of signage and speed indication only. They shall not be used as general units, for example, to define road geometry, etc.

2.3.2. Modal verbs

To ensure compliance with the ASAM OpenODD standard, users need to be able to distinguish between mandatory requirements, recommendations, permissions, as well as possibilities and capabilities.

The following rules for using modal verbs apply:

Table 2. Rules for using modal verbs
Provision Verbal form

Requirement
Requirements shall be followed strictly in order to be conform to the standard. Deviations are not allowed.

shall
shall not

Recommendation
Recommendations indicate that among several possibilities, one is recommended as particularly suitable, without mentioning or excluding others.

should
should not

Permission
Permissions indicate a course of action permissible within the limits of ASAM OpenODD deliverables.

may
need not

Possibility and capability
These verbal forms are used for stating possibilities or capabilities, being technical, material, physical, or others.

can
can not

Obligation and necessity
These verbal forms are used to describe legal, organizational, or technical obligations and necessities that are not regulated or enforced by the ASAM OpenODD standard.

must
must not

2.4. Understanding ODDs

2.4.1. What is a ODD Specification?

An ODD defines the operating conditions an ADS is designed to operate in. This includes all ranges of roads, environmental conditions, part of the dynamic elements (mainly the ego designated speed) and types of traffic participants within a macroscopic traffic during all allowed weather conditions and time-of-day restrictions. The ODD will specify the domain and its borders the ADS is designed to safely operate in or needs to be capable of handling. This opens up a multidimensional space that needs to be specified.

According to SAE J3016 an ODD is defined as follows:

SAE J3016 (2021)
Operating conditions under which a given driving automation system or feature thereof is specifically designed to function, including, but not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics.

2.4.2. Operational Domain (OD) vs. Operational Design Domain (ODD )

A specific type of terminology misuse in practice is failing to distinguish between the real world and the ODD.

  • "ODD" refers to the intended ADS capability to handle operating conditions, and not the real world.

  • "Operational Domain (OD)" refers to what the world actually is, which might (in most cases will) differ from the ODD.

Thus, when the real world (OD) is outside the intended operational design environment (ODD), the vehicle has exited the ODD and is no longer in an environment it was designed to operate within. The difference between OD and ODD highlights the limitations of the ADS. Additionally, an automated driving system feature can have only one ODD. However, a vehicle can have multiple automated driving system features. Thus it is incorrect to say, for example, that a Level 4 highway feature has "multiple ODDs of day and night with fair weather" when what should be said is that the feature has "an ODD that includes both day and night in fair weather."

In all practical cases, an ODD definition will not be exhaustive enough to cover all attributes or occurrences in an OD. Therefore, it is essential to be able to define an ODD which is not only objective but which defines clear boundaries to enable ADSs to perform fallback manoeuvres to ensure safe operation.

2.4.3. Difference between ODD and Scenario

It is important to highlight that while ODDs and scenarios are related, they are not the same. As mentioned earlier, an ODD essentially defines the operating environment for which a system is designed. It may also be seen from the perspective of the end-user (e.g. city council authority) as the operating environment in which a system should be able to operate safely. It is essential that there is an overlap between the two perspectives of the ODD, manufacturer (or the system designer) and the end-user for ensuring the safe deployment of CAVs.

A scenario defines the behavior of various actors and entities in an ODD. Below the difference between scenario and ODD is highlighted graphically. [fig-simple-odr-odd] depicts a part of an ODD with single lane non-divided road which has passenger cars. Simple scenario with passenger cars depicts a scenario which illustrates the behavior of the vehicles in the ODD of [fig-simple-odr-odd]. In summary, once the behavior of the actors is defined in a part of an ODD, it becomes a scenario. This has no influence on the fact that a scenario is a stand alone entity regardless of the ODD definition and therefore can be defined independently

600
Figure 1. Simple scenario with passenger cars
600
Figure 2. Example ODD for the domain

In Example ODD for the domain the graph on the left side represents the ontology or taxonomy the ODD specification can chose attributes from.

Figure 6a highlights the ODD attributes as per BSI PAS 1883. If an ADS has the ODD defined as Figure 6b, then the ODD comprises of a "divided single lane" road with a "T-junction" (scenery attributes of ODD). Furthermore, from the dynamic elements attributes, ODD comprises of "passenger cars". The difference between the ODD definition and the scenario definition is that the scenario definition also includes the behavior description of the actors, i.e. passenger cars.

2.4.4. The Scope and Usage of the ASAM OpenODD Language

colorfulbricks.drawio
Figure 3. ODD Definition and surrounding use cases

Being a language for defining or describing ODDs is the main purpose of the new ASAM OpenODD language, but obviously there are many more activities in the domain of automated driving system concept, design, analysis and validation that can profit from a formalized language for ODDs. The proposed ASAM OpenODD language has been designed with many of these use cases in mind, and in the present concept phase there is no reason to consider the list as final and exhaustive. Nevertheless, usability for many stakeholders, including persons who are not specialists for complex computer languages, is a key concern for ASAM OpenODD development and entails the need to keep the core language simple and not to overburden it with lots of complex language constructs just to cover each and every potential use case. The solution chosen consists of a core language that is sufficient for the primary purpose of ASAM OpenODD, which is to describe an ODD. Around this core language, many extensions are grouped as shown in the ODD Definition and surrounding use cases. Also this list does not have to be exhaustive, according to its requirements, the ASAM OpenODD language will be designed with extension mechanisms so that future extensions can still be added. As shown in the picture, even the core part of defining ODDs makes references to external work: the ontology, on which each ODD description is based, can and should be referenced from external sources (in particular OpenXOntology) and specified in their respective standardized language.

The use of all extensions (grouped around the core language in the figure) is optional, but prepared in the structure of the language (including the ability to create "bindings" of functions to 3rd party implemented functions in any language like Python or C). For the concept phase, the following list of extensions is proposed, which could technically be released as "standard libraries" such as the standard libraries of programming languages like C form a basic ecosystem, on which users of the core language can rely.

The following extensions are proposed:

TODO add that not all chapter’s are fully flushed out due to the missing working group

2.4.5. Completeness of an ODD

Since the information of the ODD is involved in the design, testing, release and so on, the completeness of the provided information becomes relevant. However, completeness should not be understood in an absolute sense, which means that no extra information can occur if a specific procedure is conducted. This is because of the open context, which makes it impossible to specify each and every aspect beforehand. Therefore, completeness is understood in a relative sense that requires us to define a reference against which the completeness of an ODD model can be measured. For the provision of such a reference, internal references as well as external sources can be used as a structure.

The completeness of the ODD results from the two aspects. This includes 1) completeness of the underlying ontology, and 2) completeness of the ODD description. To give a quick overview of the relation between the two, an ODD description language is a language format that has the capability of assigning the relevant ODD attributes to be either inside/outside its intended ODD boundary. However, these ODD attributes originate from an external taxonomy or ontology. Only by combining the ontology and an ODD description language can an ODD be produced. The BSI PAS 1883 on the taxonomy for the ODD of ADSs comments that from the perspective of the attributes/taxonomy, the ODD attributes list can never be exhaustive. This indicates that there is no 'complete' sense in terms of the underlying ontology that is used, which also resonates with one of the core language features that will be introduced in a later chapter (the extensibility of the underlying ontology). On the other hand, from the ODD language perspective, every piece of an ODD description must be complete, i.e, all the attributes from the underlying ontology need to be assigned and a binary (non fuzzy) boundary can be introduced across all the ODD attributes. To sum up, the underlying ontology can never be exhaustive, but the ODD description format needs to ensure every ODD is complete against the underlying ontology, and the permissive and restrictive concepts help to ensure completeness.

2.5. Use cases for ASAM OpenODD

In this chapter the use cases for the ASAM OpenODD are defined. In the development process the users of ASAM OpenODD were defined beforehand and the uses cases defined according to the users.

2.5.1. Users for ASAM OpenODD

Who will use and create the ODD and therefore use the OpenODD

2.5.2. User Stories

Development engineer

  1. as a development engineer, I need to understand the relevant parameters and their ranges that can occur during automated driving within the ODD and the scenarios the automated driving system might be confronted with, and be able to trace my requirements to this kind of input data.

  2. as a development engineer, I need clear criteria how to detect at runtime that I am unintentionally leaving the ODD so that I can implement detection and reaction functions for this case.

  3. as a development engineer, I need input (e.g. List) on foreseeable situations in the ODD to test my sensor setup and software algorithms appropriately before vehicle simulation and road testing even starts.

  4. as a development engineer, I desire a machine readable ODD description that support the systematic segmentation of the ODD in sub-areas where different capabilities of the AD system are needed or different grades of AD functions are safe to operate.

  5. as a development engineer I want to check tests, scenarios, recordings, etc. against the ODD description.

Test engineer

  1. as a test engineer, I would like to be able to declare an ODD to test against

  2. as a test engineer, I would like to know that a scenario that I have created is within the ODD

  3. as a test engineer, I would like to understand which element of my driving ontology/domain model is in or out of my ODD

  4. as a test engineer, I would like to know if the ODD that I have defined is well-formed and non-contradictory

the ODD is relevant for both the test track and the simulation

Data scientist

OpenODD Data Scientist Use-Cases, extending the ontology (i.e. need to support ontology by reference)

  1. as a data scientist, given a scenario library, I need to determine the parameter distribution, and coverage, so that I can assess the confidence in the results and deliver trust.

  2. as a data scientist, given a scenario library, I need to determine which aspects of the ODD are covered, so that I can assess which scenarios need to be added to support effective ODD testing.

  3. as a data scientist, given an ODD I need to determine whether a specific scenario is included in the ODD, so that I can determine whether it needs to be included in the assessment of results.

  4. as a data scientist, given an ODD, I need to determine the exposure to a specific scenario so that I can determine the weight/contribution of the scenario results in the overall assessment.

  5. as a data scientist, given two versions of the ODD I need to determine which scenarios were added or removed, so that I can determine how the analysis needs to be adapted.

  6. as a data scientist, given an ODD, I need to determine whether an occurrence is allowed by the ODD, so that I can determine whether the vehicle behaved as per the ODD specification.

  7. as a data scientist, given an ODD, I need to determine the exposure to a specific "occurrence" so that I can determine the weight/contribution of the behavior results in the overall assessment.

    • Test use-cases

Tool developer

  1. as a tool developer, I would like to have a common format that I can share to define an ODD.

  2. as a tool developer, I would like the ODD format to be formal enough that I can execute and process the containing information.

  3. as a tool developer, I would like to be able to substitute multiple driving ontologies and still use the same ODD Format.

  4. as a tool developer, I would like to test my test bench capabilities against the ODD, with the goal if the capabilities suite the ODD needs.

Scenario editor

  1. as a scenario editor, I want to create the scenarios for an ADS which are within the defined ODD of the ADS.

  2. as a scenario editor, I want to create scenarios to test the ODD attribute boundaries to test the fall-back or Minimal Risk Manoeuvre capability of the ADS.

  3. as a scenario editor, I want to create ​scenarios at different levels of abstraction.

Data annotation engineer

  1. as a label annotator I need to assemble a data library which is relevant to the ODD so the data (recordings, synthetic sensor data, etc) can be annotated accordingly

  2. as a label annotator I need to analyze data to identify ODD related situations and occurrences so that I can annotate them.

  3. as a label annotator I need to estimate the completeness of data to be annotated relative to the ODD so that users of the annotations can estimate achievable coverage.

  4. as a label annotator I need to estimate the coverage of annotations relative to the ODD so that users of the annotations can estimate coverage achieved.

Safety Engineer

  1. as a safety engineer I want to (semi-automatically) derive a situation catalog for Hazard Identification from the formal ODD definition.

  2. as a safety engineer I want to demonstrate the metrics of coverage of my considered scenarios w.r.t. the total ODD and estimate the remaining risk of hazardous scenarios within the ODD that were not covered by my safety concept and my validation approach.

  3. as a safety engineer I want to identify scenarios of unintentionally leaving the ODD from a formal definition of the limits of the ODD, in order to handle these situations using appropriate detection and reaction possibilities.

  4. as a safety engineer I want to get an understandable representation which partitions or sub-areas of the ODD exist, which failures or degradations could lead to hazards in each of them and grades of AD functions or which behaviors ensure acceptably safe operation in each of them.

Infrastructure operator (include regulators, authorities)

  1. as an infrastructure operator, I want to have easy access to ODD specifications in order to have a clear view on current and future demands on my road network

  2. as an infrastructure operator, I want to have the ability to warn vehicles in advance if they are about to chose a route through my road network which most likely is exceeding their ODD

  3. as an infrastructure operator, I want to give recommendations to automated vehicles which routes/roads/lanes to use depending on their ODD capabilities

  4. as an infrastructure operator, I want to have the ability to provide current ODD restrictions of parts of my network easily

  5. as an infrastructure operator, I want to be able to have a version controlled ODD Format

  6. as a regulator, I want to have a human-readable and exchangeable ODD format

  7. as a regulator, I want the ODD to be expandable when new regulations are introduced

3. Relations to other standards

Table 3. relations to ASAM Standards
Name of the Standard Description

ASAM OpenSCENARIO

Description and exchange format for scenario descriptions

ASAM OpenXOntology

defines an ontology (and set of attributes) for the ASAM OpenX Domain

ASAM OpenLABEL

defines a standardized format for multi-sensor data labelling and scenario tagging. The format is based on a JSON schema

ASAM OpenDRIVE

standardized exchange format for the description of road-networks and the static entities on and along the road. ASAM OpenDRIVE is based on a xsd schema, using xml as an format

ASAM OpenCRG

a binary format to define road elevation or friction profiles

ASAM ODS

allows to store information in a database context and exchange it between tools. Based on a standardized meta-model (the base model) application-domain specific data models (the application models) may be defined. This may lead to consistent data sets e.g. combining ODDs with scenario based test definitions and corresponding test results for evaluations and long term storage.

3.1. Positioning of ASAM OpenODD within ASAM activities

OpenXOverviewDomain

ASAM OpenODD is part of the ASAM simulation standards that focus on simulation data for the automotive environment. Next to ASAM OpenODD, ASAM provides other standards for the simulation domain, like ASAM OpenSCENARIO and ASAM OpenDRIVE.

ASAM OpenODD aims to describe the intended operating conditions (domain) an automated driving system is designed for. Part of the domain consists of the characteristics of the road network which can be derived from ASAM OpenDRIVE. A very simple road network is shown in Simple roadnetwork defined in ASAM OpenDRIVE. In this road network three incoming roads lead into a t-junction. This t-junction is made up out of several connecting roads.

ODD example image odr.drawio
Figure 4. Simple roadnetwork defined in ASAM OpenDRIVE

As the road network only covers parts of the static elements of the driving environment additional information source may be needed (dynamic elements). The automated driving system also needs a specification of what kind of traffic participants it needs to be capable of handling. These dynamic elements and their behavior can be represented using ASAM OpenSCENARIO. A simple scenario with 4 passenger cars is shown in Simple scenario defined in ASAM OpenSCENARIO.

ODD example image osc.drawio
Figure 5. Simple scenario defined in ASAM OpenSCENARIO

When analyzing the above two examples a few attributes of the ODD can already be identified. The automated driving system shall be designed to operate in:

  • divided single lane roads, with right hand traffic

  • a t-junction without traffic lights

  • passenger cars

This is shown in Example ODD for the domain

ODD example image odd.drawio
Figure 6. Example ODD for the domain

In addition to ASAM OpenSCENARIO and ASAM OpenDRIVE, ASAM OpenODD is also closely linked with the ASAM OpenLABEL standard, especially on the scenario tagging aspect. Figure below illustrates one example use case of the scenario tagging within ASAM OpenLABEL, it can be seen that each scenario is associated with corresponding tags from a bucket of tags. Based on such information, scenarios can be searched easily within the database. The link between ASAM OpenODD and the ASAM OpenLABEL scenario tagging aspect is that ODD tags form a major part of this 'bucket of tags'. The ASAM OpenODD attributes are closely aligned with the ASAM OpenLABEL tagging model. By utilizing such common properties, organizing and searching of scenarios and ODDs, and their dependencies can be effectively achieved.

OpenLABEL tagging example
Figure 7. OpenLABEL scenario tagging use case

3.2. References to other standards

Name of the Standard Description

ISO 34503 (in development)

Provides a taxonomy for operational design domain for automated driving systems, along with a high level definition format for use by regulators and non-coders.

BSI PAS 1883

Provides a taxonomy that can be used to ODDs for automated driving systems

SAE AVSC00002202004 AVSC Best Practice for Describing an Operational Design Domain: Conceptual Framework and Lexicon

Aims to establish a best practice for Operational Design Domain description, establishing commonly defined terms and recommending a framework in which they can be applied.

SAE J3016 (2021) / ISO 22736

Defines the term ODD and its use for classifying levels of autonomy for automated driving systems

ISO 21448 SOTIF

Provides guidance on defining the safety of an ADS system or feature for its intended functionality.

ISO TS 5083 (in development)

Provides guidance on safety of an ADS system (design and verification stage).

4. Language and Format for ASAM OpenODD

The language and format chapter forms the core language concepts of the OpenODD. Every language development was made against a pre-identified requirement which was identified during the initial phase of the project. The content in this section will provide an overview of the overall development process, followed by the identified requirements clusters, and furthermore examples were provided for each of the requirements. These examples contain two different syntax proposals and the query semantics (where applicable). Thirteen different requirement clusters in total have been identified, which define the scope of the OpenODD format:

  1. readability

  2. composability

  3. conditional statement

  4. extensibility of ontology

  5. binary boundary

  6. class/metric/data type/ units

  7. expression/abstraction

  8. extensibility by function

  9. operators

  10. permissive/restrictive

  11. language extensions(probability

  12. uncertainty)

  13. integration with scenario-based testing workflow.

From the thirteen requirement cluster examples, both the semantic and syntactic aspects of the language are covered.

The chapter includes:

  • Overview of the language concept development process

  • Requirements collection and requirement clusters creation

  • Main language concepts

  • Main language concept (extensibility of ontology) and extended language concepts (metrics, probability&uncertainty and extensibility by function) centered around the extensibility nature of the language

  • Integration with scenario-based testing workflow

4.1. Example syntax representation

During the concept project several syntax approached evolved and were proposed. The project decided to move forward with two of these proposal:

  1. Syntax 1 is aligned with OpenSCENARIO 2

  2. Syntax 2 aligns with the query semantics approach which evolved during the OpenODD concept project phase

4.1.1. Introduction into syntax 1

Modelling assumptions and principles for representing and using ODDs using OSC 2.0 language:

OpenSCENARIO 2.0 is an object oriented, aspect oriented, declarative, domain specific programming language. As such, it offers the full feature set of programming languages, in terms of operators, types and variables. It includes methods, structures, units, constraints specification and more. The examples are using OpenSCENARIO 2.0 syntax and constructs, with some assumption on a potential way to include and refer to ODD specification, in the same language in which scenarios are written, and using the same types, units and entities.

ODD definition:

We assume the full ODD ontology is defined in a type called ODD.

Sets of values are defined as enumerated types, for example:

type rain_type: [none, light, moderate, heavy]
type wind_type: [calm, moderate, storm]
type road_type: [city_road, motorway]

Sub-hierarchies are defined as structs, for example:
struct weather_type: odd_struct:
    rain: rain_type
    wind: wind_type

The full ODD is defined by hierarchical composition:
struct ODD: odd_struct:
    weather: weather_type
    road: road_type
and so on…
odd_struct

ODD types are declared using a subtype of OpenSCENARIO struct called odd_struct. odd_struct fields may represent:

  • a physical value or range like speed

  • an enumerated value representing some classification, like light_rain

  • a Boolean flag like has_guardrail

  • a numeric value or range like number_of_lanes

  • a sub odd_struct representing a hierarchy

The current ODD is captured by top.odd which is an instance of odd_struct

constraints

Constraints are added to odd_struct, representing the limitations imposed by the ODD specification. They should be set-oriented. Only rarely does constraining to a specific value makes sense in ODD context. The synthetic constraint all_default_values() restricts all unconstrained fields to their default values, , implying that any restrictions on them will lead to contradiction. This is used when defining a restrictive ODD.

OpenSCENARIO 2.0 supports importing of packages (source files). Once imported, the types defined in these packages are available. This is the expected method to import either the ODD ontology/taxonomy , or ODD definitions.

Binding method to external library

The binding of a function defined in an external library to a method in OSC2.0 is done as follows:

def <method-name>(<arguments>):<returned-type) is external.cpp("function-name", "shared-object-name")

Once defined the method can be called to invoke the associated library function.

4.1.2. Introduction to syntax 2

The semantics of an ODD specification are defined as a mapping of all possible variable value assignments (continuous, integers, binary or enumerated type) to a True/False determination. Intuitively, it is equivalent to an SQL query selecting of all possible situations whereby the "WHERE" condition is provided by the ODD specifications: Situation returned by such query are included in the ODD; all other situations are excluded.

Each statement must start at a new line, but it may span multiple lines. When a statement spans multiple lines, the 2nd and subsequent lines must be indented. Each statement must start with one of the following keywords: INCLUDE, SUITABLE, UNSUITABLE, DETERMINE, ADDCOND,MEASURE. In addition, the TAG keyword can be used towards the end of certain station as a packaging mechanism to bundle a set of ODD attributes and form under a new ODD name. The keywords defined are not yet completed and need to be extended for cover all required functionalities. Each statement is associated with specific syntax rules:

  • INCLUDE - Include statements start with the keyword "INCLUDE" followed by a file name. Included files may be an ontology file or other ODD specifications, and there is no restriction with regard to the depth (or breadth) of the include tree.

  • SUITABLE/UNSUITABLE - these two keywords are used at the start of the corresponding statements to indicate the permissive/restrictive nature applied to each individual statement. They have the following patterns (parenthesis denote optional keywords):

    suitable condition
    SUITABLE <concept> (EXCEPT) WHEN <condition>
    unsuitable condition
    UNSUITABLE <concept> (EXCEPT) WHEN <condition>
  • ADDCOND - it is used to add custom extension to an existing domain model or ontology, for example if user wants to add 'my_special_T-junction' to an existing ODD attribute 'T-junction', ADDCOND can be used. The standard format for using ADDCOND is: ADDCOND <new_condition> TO <concept> (AS <relationship>) whereby:

    • <concept> is defined by a previous statement, either in the same file or in an included file.

    • <relationship> defaults to "child-of" if the expression is omitted; otherwise, a specific relationship is added to the concept graph, this part is optional. Example: user would like to add 'my_special_T-junction' under the T-junction class, the format would be:

ADDCOND my_special_T-junction TO T-junction
  • DETERMINE - this keyword is used to construct new classes/or to assign existing classes using a set of values. For example, is user wants to specify that 'my_special_rain' is when both the rainfall rate greater than 5mm/hr and the droplet size greater than 1mm diameter, the keyword DETERMINE can be used to perform such assignment (see in the example below). The format follows the structure: DETERMINE <concept> WHEN <expression> whereby the <expression> is an and-or-tree converting continuous variables/metric values to True/False values.
    For example:

    define a constraint with a given range
    DETERMINE rain_heavy WHEN (rain_rate >5mm/hr) AND (droplet_size >1mm)
  • MEASURE - this is used to assign measuring unit and data type to an ODD attributes. For example, to assign mm/hr as a measuring unit for rainfall rate, while the datatype is float, such operation can be achieved by using this keyword. The "MEASURE" statement follows the format as below :

    • MEASURE <symbol> TYPE numeric | enum If TYPE is "numeric" we have the following additional component to this statement: MEASURE <symbol> TYPE numeric UNITS <unit_symbol> [RANGE <min>,<max>] [DEFAULT <value>]

For the example listed above, the MEASURE statement would be

+

MEASURE rainfall TYPE float UNITS mm/hr
  • TAG - TAG is used to package a selected numbers of parameters and provide them with a new ODD name. For example if user need to combine ODD1 and ODD2 to form a new ODD3, this keyword can be used. It follows the format below: <conditions> TAG <new_ODD_name>, where the conditions is the inclusion or exclusion. An example is shown below:

    SUITABLE ODD1 AND ODD2 TAG ODD3

4.2. Overview

This chapter focuses on the format of the OpenODD language. As illustrated earlier, the OpenODD language needs to satisfy various requirements from different target users. Such requirements will form the foundation of the language development, and will create a coherent storyline throughout this chapter.

To understand how this chapter is structured, it is important to first understand the overall development process the work package took during the project. In the early stage of the development, this list of requirements went through an expansion followed by a contraction phase. The expansion part built on top the identified use cases, and called for a further gathering of additional relevant requirements within the working group, this resulted in an extended requirement list. The group then evaluated the relevance of each submitted requirements, and provided necessary examples. The contraction part performed a clustering exercise on this extended list, many similar requirements were grouped together to form a 'requirement cluster', whose definition is the union of all its sub-requirements. Upon obtaining a list of requirement clusters, language concepts centred around each cluster were developed and proposed. Due to the nature of a concept project, the group decided to accept all the relevant proposals and suggestions (providing that the same concept proposal covers all the clusters). It will then be the responsibility of the standardization phase to consider the pros and cons of each concept and arrive at an unified concept. In addition to the language concept proposal, the group also utilized a more semantic-centred approach to illustrate each cluster, this is based on a neutral concept of query semantics, which is represented in syntax 2.

With the overall workflow introduced, readers should find the chapter structure easy to follow. Section 4.2.1 provides the background and reference to the extended requirement list, the list also indicates group members' comments and those requirement entries that the group decided to be irrelevant. Section 4.2.1 then introduces an overview of the requirement clusters together with their definitions which resulted from the extended requirement list. Section 4.3 onwards provides detailed content associated with each of the clusters.
Such content includes:

  1. a detailed definition of the cluster,

  2. example challenge(s) the group proposed that can represent the cluster,

  3. how the challenges can be tackled using the OSC2.0 language and query semantic,

  4. how such challenge(s) can be tackled with different proposed and developed syntax.

By reading this chapter, readers will gain understanding of the scope of the language concept, the details of each requirements, and the approaches/proposals.

4.3. Requirements for ASAM OpenODD

4.3.1. Collected requirements list

During the runtime of the project the project group collected requirements, the requirements are clusters into categories. In total the categories are as follows

syntax

rules of writing, Format

semantics

how to interpret the written

process

how to integrate and/or use the written

The link above leads to the extended list of requirements which form the foundation of future OpenODD language & format development work. Please note that within this list, project members have voted that requirements 60, 56a, 56b, 54, 62, 61, 37 will not be considered for the first draft of the OpenODD Format.

4.3.2. Requirement Clusters Overview

The Requirements are being clustered in the following clusters

Nr.

Cluster

Description

CL1

Human readability

ODD language needs to be intended to be used by various stakeholders, and it will be used as input to testing and analysis as well as specification, communications, analysis and debugging.

CL2

Machine readability and query

The language needs to be machine readable, can be parsed using any text parsers, should enable queries for attributes of the ODD. UTF8 Test using DSL,XML,YAML,others.

CL3

Composability

The language shall support combining/re-using multiple ODD definitions into a new, wider ODD, and it shall provide the ability to create a named subset of ODD attributes

CL4

Parameterization and Templating

In order to enable flexibility in definition, and flexibility in usage for testing and definitions, as well as the ability to supply templates for ODD definitions for various usages , the language should enable a parametrization mechanism, where names are replacing values, and are resolve toward actual usage time.

CL5

Conditional statement

Language specification needs to support conditional statement or reduced ODD. This can be tackled by imposing constraints on the full operational range of specific attributes.

CL6

Extensibility of Ontology

The language shall support the addition of new attributes and values that are not in the original ontology. It shall support the establishment of standard libraries, e.g. PAS 1883.

CL7

Binary boundary

The language shall map every situation into binary in/out of the ODD, convert 'uncertain' or 'fuzzy' situation into binary.

CL8

Class metric / datatype / units

  1. class attributes must be separate from metrics, eg. the “rain” yes/no is separate from “droplet size”, the "speed limit”<100kph is separate from “road” class

  2. need to handle different datatype, enum, float, boolean, etc

  3. defining the unit is compulsory, unless it is a unit-less ratios.

CL9

Probability / Uncertainty / risk

The language shall support qualifiers to represent uncertainty,probability and risk as optional element

CL10

Class Hierarchy / expressiveness / abstraction

The language format needs to be able to express the class hierarchy provided by the attribute group (WP01), it needs to support inheritance, be consistent with the class categorises (keywords), data type and units. It shall support abstract ODD definitions by choosing different hierarchical levels.

CL11

Extensibility by function

The language shall be extensible by internal or external functions.
Basic mathematical functions (like sqrt(), max(), min() ) or functions with respect to the application domain (e.g. distance() ) are required for conditions, limits and metrics. They can come from supplied library or external binding (e.g. C, Python)

CL12

Operators

The language shall use a set of mathematical and logical operators, such as +, -, and, or, >, <, etc

CL13

Permissive / Restrictive

The language should take a mixed permissive and restrictive approach to its statements of what is inside the ODD.
Use explicit statements for inclusion or exclusion. Allow specifying exceptions while guarding from changing scope

CL14

Integration in scenario based testing workflow

The language shall supply the ability to refer to ODD definition from scenario definition, interpretable to create a full test description , and can be used for different purposes (such as requirements, checking, success criteria or testing)

CL15

Compatible with OpenX Standards

The language shall be compatible with other OpenX standards and ASAM ODS, share a common domain model.

4.4. Concepts for the language and format of ASAM OpenODD

Of the 15 requirement clusters, some of them can be bundled together due to similar nature. Instead of mechanically introduce them one by one, the following sections will be divided into three main parts: section 4.3 contains the first group of clusters where little similarity can be drawn between them, in addition, the concept of mandatory and optional attributes will also be covered in section 4.3; section 4.4 contains those clusters with extensibility property, this includes those belong to the core language concepts (i.e, extension by function, extensibility of ontology), and also those outside the core language concepts (i.e, metrics and probability); section 4.5 will introduce the integration with the scenario-based testing workflow.

4.4.1. Mandatory vs optional Attributes

The role of mandatory and optional attributes can be illustrated in combination of another key language feature (permissive/restrictive, which will be introduced in Permissive and restrictive definition). To briefly introduce, permissive means everything that is not explicitly describe in the ODD will be permitted, whereas restrictive corresponds to hte opposite. Now given the following ontology

OntologySceneryRoadNetworkUrban RoadsLeft hand trafficLane dimension (e.g, <2m, 2-3.5m, >3.5m)ON_OFF rampHighway_exit_entryDynamic ElementsTrafficspeed = {minimum_speed = 0.0 kph, maximum_speed = 60.0 kph}traffic_density = {low, medium, high}traffic_participants = {!Lorries}WeatherSunnyFoggyRainySnow
Figure 8. Ontology

If a blank ODD is defined in permissive mode, all the above attributes are allowed, if the blank ODD is defined in the mode restrictive, none of the above attributes would be allowed. This means that the only required attributes of an ODD defined using ASAM OpenODD are the global permissive/restrictive mode, and the reference to the underlying ontology. However, although not mandatory in nature, in some cases the attributes which can be associated with numerical values are also recommended to be specified. In the above example, the lane dimension is one such example. Let us imagine an ODD with such an ontology and a global permissive mode, this means the system can handle any lane dimensions that are listed in the ontology. While this could be true it is unlikely an ADS with a conventional vehicle size can handle extreme narrow lanes such as 1.50m.

4.4.2. Human and machine readability

ODD specifications are intended to be shared across wide range of stakeholders such as regulators and test engineers, it would be used as input to testing and analysis, but at the same time to be used by humans for purposes of specification, communications, analysis and debugging of systems under development. This requires ODD specification to be understandable to both technical and non-technical users. In the meantime, ODD specifications should enable machine readability for purposes such as runtime ODD coverage checking, comparisons between multiple ODDs. The language is intended to serve as input to computer programs, that should be able to parse, analyze , query and interpret the ODD information. In particular, query has been identified as an important requirement within the project group. Query could be performed between a traffic situation (corresponding to the concept of OD mentioned in the introduction) and the corresponding ODD specification utilizing an ontological approach.

Example use case:

An example ODD states that motorway is only suitable when there is no rain, up slope is not suitable as the vertical geometry.

Example representations:

  • Syntax 1:

    keep(road_type == motorway => odd_5.weather.rain == none)
    keep(geometry.vertical != up_slope)
  • Syntax 2:

    SUITABLE Motorway EXCEPT WHEN Rain
    UNSUITABLE Up_slope

4.4.3. Composability

The ASAM OpenODD language format should be composable, (i.e., combine ODD definitions into a new, wider one or use inheritance mechanism). In addition, the ability to re-use a defined section of an ODD should also be possible, by providing the ability to create a named subset of ODD attributes that can be defined together. Unions and intersections are examples of composing two ODDs, in addition the negation of ODDs can also be composed with other ODDs. When selecting subclasses from a parent class, the dot operator can be used such that if b is a sub-class of a, then a.b will point towards the b subclass, such convention can be seen in both syntax examples below. From a metric perspective, the language shall allow building new metrics out of existing metrics, functions and operators. This will be covered in the extensibility section.

The ability of being composable will also enable comparison of ODDs and identfiying contradicting specifications in the ODD. This can be very helpful when comparing a vehicle ODD to a City ODD, checking wether the car is able to operate safely in the streets of the city.

In the following abstract example the underlying meaning of of composing ODDs is illustrated. As can be seen that the first tree of attributes (first ODD) differs from the second tree, by composing these two together the third tree (ODD) is obtained, which is the union of the two. Please pay attention to the conditions that the speed for highway and for urban roads are separate (these are inherited from the individual ODDs and are not mixed during the composing process).

ODDSceneryRoadNetworkHighway2 Lane each directionLeft hand trafficON_OFF rampHighway_exit_entryDynamic ElementsTrafficspeed = {minimum_speed = 0.0 kph, maximum_speed = 130.0 kph}traffic_density = {low, medium}WeatherSunny
Figure 9. first ODD
ODDSceneryRoadNetworkUrban RoadsLeft hand trafficDynamic ElementsTrafficspeed = {minimum_speed = 0.0 kph, maximum_speed = 50.0 kph}traffic_density = {low, medium}WeatherSunnyRainy
Figure 10. second ODD
ODDSceneryRoadNetworkUrban RoadsLeft hand trafficHighway2 Lane each directionLeft hand trafficON_OFF rampHighway_exit_entryDynamic ElementsTrafficspeed @ Urban Roads = {minimum_speed = 0.0 kph, maximum_speed = 50.0 kph}speed @ Highway = {minimum_speed = 0.0 kph, maximum_speed = 130.0 kph}traffic_density = {low, medium}WeatherSunnyRainy @ Urban Roads
Figure 11. composed ODD

In the following abstract example the concept of comparing ODDs is illustrated. The mentioned "City ODD" can be seen as requirements towards the capability of the ADS. ADS ODD will be the designed capability of the vehicle. By comparing the two ODD specifications the suitability of the ADS towards the City requirements can be established.

ODDSceneryRoadNetworkUrban RoadsLeft hand trafficON_OFF rampHighway_exit_entryDynamic ElementsTrafficspeed = {minimum_speed = 0.0 kph, maximum_speed = 60.0 kph}traffic_density = {low, medium, high}traffic_participants = {!Lorries}WeatherSunnyFoggyRainySnow
Figure 12. City ODD
ODDSceneryRoadNetworkUrban RoadsLeft hand trafficDynamic ElementsTrafficspeed = {minimum_speed = 0.0 kph, maximum_speed = 50.0 kph}traffic_density = {low, medium}traffic_participants = {!Cyclist}WeatherSunnyRainy
Figure 13. ADS ODD
ODDSceneryRoadNetworkUrban RoadsLeft hand trafficDynamic ElementsTrafficspeed @ City = {minimum_speed = 0.0 kph, maximum_speed = 60.0 kph}speed @ Vehicle = {minimum_speed = 0.0 kph, maximum_speed = 50.0 kph}traffic_density @ City = {low, medium, high}traffic_density @ Vehicle = {low, medium}traffic_participants @ City = {!Lorries}traffic_participants @ Vehicle = {!Cyclist}WeatherSunnyFoggy @ CityRainySnow @ City
Figure 14. compared ODD

Result:

The City ODD spedifies that any vehicle there (if going in automated mode) shall expect and be capable of handling other vehicles moving with up to 60kph while the ODD specification of the automated vehicle states under development states that its own maximum speed is 50kph. This alone is not a problem for operating the vehicle in the city (ego vehicle speed capability and traffic speed capability are two different things); it is sufficient that the vehicle can cope with traffic up to 60 kph, even if it cannot reach that speed itself.

Be aware that case would be different if there were constraints on the achievable speed of our vehicle, for instance related to a highway settings, where in some countries traffic regulations require that only motor vehicles with a nominal max speed of at least 60kph are allowed to be present; then, a vehicle with a technical max speed of only 50kph would be explicitly forbidden to enter that highway.

But there are more differences between our vehicle’s ODD and the city ODD:

  • The vehicle can only handle up to medium traffic density. but the city’s ODD specifies that there can be high traffic density.

  • The vehicle’s ODD specification states that is not capable of handling cyclists, where the city only states that lorries are not to be expected - so cyclists can occur.

  • In terms of weather, the gap is that the vehicle can not operate in fog and snow which is according to the city ODD is to be expected.

As a result it can be concluded that the specified vehicle is not fully capable of operating in the given ODD of the city, and thus is not allowed to enter the city on automated driving (or, respectively, allow the human user to switch on the automated driving function while running in that city).

Note: In the section on queries there are more usage examples of such comparisons between a vehicle’s ODD and other ODDs, roadmaps or scenario sets.

Example use case: Compare and Combine

Given an ODD_1 which specifies that motorway is not suitable without restriction on the remaining classes on the ODD attribute hierarchy. Given another three ODDs (ODD_2, ODD_3, ODD_4) which specify wind conditions, rainfall condition and snowfall condition. The goal is to:

  1. construct a weather ODD (ODD_5) by combining ODD_2, ODD_3 and ODD_4

  2. construct a combined ODD (ODD_6), which takes the motorway ODD and weather ODD.

  3. in addition, illustrate how to obtain the intersection, union and difference between two ODDs

    • Example syntax:

      • Syntax 1:

        # ODDs mentioned in challenge:
        odd1: ODD:
            keep(road != motorway)
        odd2: ODD:
            keep(weather.wind != storm)
        odd3: ODD:
            keep(weather.rain in [none, light])
        odd4: ODD:
            keep(weather.snow == none)
        
        odd5: ODD:   # feature selection
            keep(weather.wind == odd2.weather.wind and
                 weather.rain == odd3.weather.rain and
                 weather.snow == odd4.weather.snow)
        
        # Since odd2, 3, 4 are orthogonal ( no overlap )  we could also create a union:
        odd5_1: ODD: # union
            keep(weather == union(odd2.weather, odd3.weather, odd4.weather)
        
        odd6: ODD:
            keep(road == odd1.road)
            keep(weather == odd5.weather) # recursive equivalence
      • Syntax 2:

        # For combining ODD1 and ODD2 into ODD3, various options depending on the combination operation:
        
        INCLUDE example_ontology # this contains rain_type, wind_type, road_type, geometry, weather_type
        
        ADDCOND [none, light, moderate, heavy] TO rain_type
        ADDCOND [calm, moderate, storm] TO wind_type
        ADDCOND [city_road, motorway] TO road_type
        
        # ODD ontology modeled as a hierarchical composition of odd_structs
        ADDCOND [rain_type, wind_type] TO weather_type
        
        SUITABLE motorway EXCEPT WHEN rain
        UNSUITABLE geometry WHEN up_slope
        
        # ODDs mentioned in challenge:
        UNSUITABLE road_type WHEN motorway TAG ODD1
        UNSUITABLE weather_type WHEN storm TAG ODD2
        SUITABLE rain_type WHEN none OR light TAG ODD3
        UNSUITABLE weather_type WHEN snow TAG ODD4
        
        SUITABLE * WHEN ODD2 AND ODD3 AND ODD4 TAG ODD5
        SUITABLE * WHEN ODD5 AND ODD1 TAG ODD6
        
        # In addition, below illustrate different composabilities
        # Intersection:
        SUITABLE * WHEN ODD1 AND ODD2 TAG ODD3
        
        # Union:
        SUITABLE * WHEN ODD1 OR ODD2 TAG ODD3
        
        # Difference ODD1-ODD2:
        SUITABLE * WHEN ODD1 AND NOT(ODD2) TAG ODD3

Example use case: selective combination

Import a sample ODD as ODD1 which constraints on both road type and weather. Import another ODD (ODD2), construct a new ODD (ODD3) that contains the weather part of ODD1 and the whole of ODD2.

Example syntax:

  • Syntax 1:

    import odd_lib/ODD1.osc-lib     # declares ODD1
    import odd_lib/ODD2.osc-lib     # declares ODD2
    
    odd1: ODD1 # as imported
    odd2: ODD2 # as imported
    odd3: ODD
        keep(me.* == odd2.*) # all features equal to odd2
        keep(weather == odd1.weather) #select the weather sub-hierarchy
  • Syntax 2:

    # It is important to differentiate ODD and ontology, within the statement
    # attribute without namespaces can be treated as coming from the ontology.
    # We could use namespaces for include multiple ontologies.
    INCLUDE PAS_1883.owl
    INCLUDE sample.odd AS ODD1
    INCLUDE ODD2
    SUITABLE ODD1.weather AND ODD2 TAG ODD3
Combining ASAM OpenODD files should not allow inconsistencies in the combined ODD description. Checkers and validators should be implemented in the toolchain, to avoid any inconsistencies.

4.4.4. Parameterization and Variable definition

In order to enable flexibility in definition, flexibility in usage for testing as well as the ability to supply templates for ODDs, the language should enable a parametrization mechanism, where actual values are replaced by variable names, and are parameterized during actual usage time.

Example use case: Variable usage

Example on how to use variable top_speed_limit in ASAM OpenODD

Example syntax:

  • Syntax 1:

    keep (speed < top_speed_limit)
    # where the value of top_speed_limit is undefined, and will be defined once this statement is actually being instantiated or used
  • Syntax 2:

    DETERMINE safe_speed WHEN speed < top_speed_limit
    SUITABLE * WHEN safe_speed

Example use case: Variable definition

Example on how to use variable rain of type enum in ASAM OpenODD, the actual value of the enumerations can be defined during usage

Example syntax:

  • Syntax 1:

    type rain_type: [none, light, moderate, heavy]
  • Syntax 2

    # define the constraint for rain, a constraint will either be True or False
    ENUM    rain VALUES {no:     WHEN mNoRain,
                         low:    WHEN mLowRain,
                         medium: WHEN mMediumRain,
                         heavy:  WHEN mHeavyRain};

4.4.5. Conditional statement

Language specification needs to support conditional statement or reduced ODD. This can be tackled by imposing constraints on the full operational range of specific attributes.

When defining an ODD for a vehicle or a town, it might be necessary to define constraints according to certain conditions. This might be due to environmental conditions that degrade the capability of the vehicle or geographic specific conditions that have an impact on defined constraints.

Example use case: basic constraint definition

Examples showing how basic constraints are handled.

Example representations:

  • Syntax 1

    odd1: ODD:
        keep(weather.rain in [heavy] => speed in [0..60]kph)
        keep(weather.rain not in [none, light] => road != motorway)
  • Syntax 2

    SUITABLE rain.heavy WHEN speed < 60
    UNSUITABLE Motorway EXCEPT WHEN rain.none OR rain.light

Example use case: add constraint definition

Adding constraint to an existing constraint

Example representations:

  • Syntax 1

    odd1: ODD:
        keep(speed in [0..80]kph)
        keep(weather.fog == heavy_fog => speed in [0..40]kph)
  • Syntax 2

    adding an additional Condition in Syntax 2
    SUITABLE speed > 10 EXCEPT WHEN rain.heavy
    SUITABLE rain.heavy WHEN (speed < 60 AND speed > 10)

Example use case: Apply basic constraints

The velocity in which an ADS system is operating is defined to be between 0 and 80kph. However, in the case of the a heavy fog ( assuming heavy fog is well defined ) , the ADS system is constrained to operate in velocity range of 0 to 40kph.

Example representations:

  • Syntax 1:

    struct any_odd:
        max_speed:  speed
        weather: odd_weather
        keep(max_speed < 80 kph) # restricted unconditionally
        # in case of fog, restrict further:
        keep(weather == heavy_fog => (max_speed <= 40 kph))
  • Syntax 2:

    DETERMINE speed_high WHEN speed (> 40 AND =< 80)
    DETERMINE speed_low WHEN speed (=< 40)
    # MEASURE for specifying unit can also be combined with DETERMINE
    MEASURE speed UNITS kph
    UNSUITABLE speed_high WHEN heavy_fog
    SUITABLE speed_safe AND speed_low EXCEPT WHEN heavy_fog

Example use:

Given an ODD hierarchical tree, we would like to express that Motorway is only suitable when there is no Rain (conditional ODD statement), within the Geometry, Up-Slope is not suitable, all Weather conditions are suitable.

Example representations:

  • Syntax 1

    odd1: ODD:
        keep(weather.rain != none => road != motorway)# no motorway in rain
        keep(geometry.vertical != up_slope)
  • Syntax 2:

    SUITABLE Motorway EXCEPT WHEN Rain
    SUITABLE Vertical_geometry EXCEPT WHEN Up_slope

4.4.6. Binary boundaries

Any ODD definition requires the ability to have binary decision on any constraint, specified in the ODD.

The language will define the boundaries of the ODD specification. The language shall allow the mapping of every ODD terminal element into either inside or outside of the ODD in a binary manner by any external processor. This results in an ODD representation which can be evaluated to a set of TRUE/FALSE ODD terminal statements.

From a mathematical point of view such a representation relates to the ordered binary decision diagrams (OBDDs). From a programming point of view the OBDDs can be handled through boolean logical programming that allows to efficiently apply to OBDDs many logical operations (like conjunction, disjunction and negation) promoting composability of the language

Imposing restrictions to the possible combinations of ODD elements (concurrent evaluation of specific ODD elements to TRUE) can be achieved through conditional statements

In the envisioned (declarative) ODD domain specific language an ODD definition will be a collection of statements. This can be achieved by introducing to the language the following type of statement

propositional constraint statements: A statement specifying a propositional constraint, using a logical expression over previously defined variables, which can be evaluated using "truth tables", i.e. each ODD sub-statement as well as any composite ODD statement can be evaluated to either TRUE or FALSE. The semantics of propositional constraint statements define the mapping of every possible combination of values for its input expression statements to a True/False value. Propositional constraint statements_statements may be defined over other constraint statements.

Example use cases:

A road is considered unsafe for the automated driving system, if there is heavy or medium rain.

Example representations:

  • Syntax 1

    keep(rain not in [medium, heavy])
  • Syntax 2:

    SUITABLE rain EXCEPT WHEN medium OR heavy

4.4.7. Operators

linked to requirement cluster [CL12]

The operators must match the syntax of the chosen language, for example the operators of range and others should match the encapsulation language

The following operators will be build into the OpenODD Format:

Operator

Description

Example

Comment

+

plus, Addition

x+y

will return the added value

-

minus, Subtraction

x-y

will retrun the subtracted value

*

multiplicator, Multiplication

x*y

will return the multiplicated value

/

divider, Division

x/y

will return the divided value

%

modulo, Modulus

3%2 = 1

will return the modulo value

//

Floor division

x//y

!

negation, equivalent to "NOT"

!(y<5)

=

asign

x=y

will assign a value to a variable

==

equal

x==y

will return a bool

!=

not equal

x!=y

will return a bool

&

equivalent "AND"

x AND y

will return true if x and y is true

|

equivalent "OR"

x OR y

will return true if x or y is true

<

less

x<y

will return a bool

less equal

x⇐y

will return a bool

>

greater

x>y

will return a bool

>=

greater equal

x>=y

will return a bool

in

element in sequence

x in [a,b,c]

will return a false

not in

element not in sequence

x not in [a,b,c]

will return a true

4.4.8. Hierarchy

The language format needs to be able to express the class hierarchy provided by the attribute group (WP01), it needs to support inheritance, be consistent with the class categories (keywords), data type and units. It shall support abstract ODD definitions by choosing different hierarchical levels.

Example use cases: Inheritance

Challenge: Define a restrictive ODD that specifies motorway as the allowed road type, and junctions are not suitable. In addition, everything else that is not mentioned is not allowed. In this case, when the hierarchical class 'Junctions' is used, the restriction will automatically inherited by all its sub-classes, this is similar to the next abstraction example.

Example representations:

  • Syntax 1

    import PAS_1883  # Assuming this includes PAS1883 fully elaborated ODD
    extend junctions_type:      # No junctions of any sub-type allowed
        keep(size == 0)
    
    restrictive_odd: ODD:
        keep(road_topology.drivable_area.road == motorway)
            # only motorways allowed
        keep(all_default_values())
           # restrictive, only explicitly included values allowed
  • Syntax 2:

    MODE Restrictive
    INCLUDE PAS_1883.owl
    SUITABLE motorway
    UNSUITABLE junction # this unsuitability is automatically inherited by all the types of junctions under the junction class

Example use cases: Abstraction

Abstraction allows the user to define the ODD at any level of abstraction corresponding to the Ontology

Define a ODD that allows all dynamic elements defined in the ontology, except when there is any type of rain. In this example, ODD can be specified at the 'Dynamic elements' level, the semantic echos with the inheritance property which means all the sub-classes of 'Dynamic elements' are also applied.

Example representations:

  • Syntax 1

    import PAS_1883  # Assuming this includes PAS1883 fully elaborated ODD
    
    no_rain_odd: ODD
        keep(dynamic_elements.weather.rain == none) # Disallow rain
  • Syntax 2:

    SUITABLE Dynamic Elements EXCEPT WHEN rain

4.4.9. Permissive and restrictive definition

Permissive/Restrictive modes can be considered in two levels — at the individual statement level, and at the global level.

At the individual statement level, the language should take a mixed permissive and restrictive approach within its statements in order to be flexible when defining what is inside/outside the ODD (using explicit statements for inclusion or exclusion along with conditionals). The user of the language is responsible when specifying multiple permissive/restrictive expressions to guard from changing scope.

At the global level, restrictive mode is recommended to be considered such that ODD attributes that are not defined with statements can be evaluated by default to NOT allowed. This is particularly handy in the usual case where your custom ODD definition is based on an existing ontology (see importing an existing ODD ontology at the beginning of your new ODD definition) or when expressing a narrow ODD. Below is a reference from MUSICC (Multi User Scenario Catalogue for Connected and Autonomous Vehicles) project which defines the global permissive/restrictive. Design considerations for ODD ontology by CATAPULT, March 2020. Users can choose to define an ODD using permissive at the global level where ODD attributes that are not defined with statements can be evaluated by default to allowed. This can be useful in expressing wide ODDs.

"Conditions may exist in the real world which are not modeled in the ODD ontology, and therefore cannot be represented in an ODD specification. With a restrictive ODD definition, it is clear that these are outside of the ODD, which is consistent with the SAE J3016 definition(since the ADS is unlikely to have been “specifically designed to function” with an unforeseen feature). If the ODD definition is permissive, it is unclear whether the element is allowed for the vehicle or not. […​]For a hierarchical structure to be useful, it should be possible to use combined expressions (e.g.“Rural roads except motorways”); otherwise, all but the simplest of ODDs require every leaf item to be specified."

ODD statements' permissive/restrictive mode should not be confused with natural language permissive/restrictive classifications (i.e. in the natural legal language "The court allows act x" is classified as permissive language while in the ODD language the same expression denotes a restrictive mode).

Example use cases: ODD definition for a T-Junction

For the example t-junction represented in the image and ontology below. Lets consider what permissive and restrictive means at a global level. For this we will have no 'local' statements so that we can visualize the effects of the 'global' permissive/restrictive.

Using a restrictive base state. A 'blank' ODD definition for the attached t-junction ontology would mean that all leaf attributes can be considered as outside of the ODD: restrictive - any attributes that are not mentioned in the ODD description but are present in the ontology (T-junction ontology) are assumed outside of the ODD’s scope.

Using a permissive base state. A 'blank' ODD definition for the attached t-junction ontology would mean that all leaf attributes can be considered as inside the ODD: permissive - any attributes that are not mentioned in the ODD description but are present in the ontology (T-junction ontology) are assumed inside of the ODD’s scope.

Note that in both examples ideally attributes that require numeric input would be separately defined, as these could not be simply included/excluded. i.e lane dimensions should be defined regardless of base state for clarity of description.

T junction
Figure 15. example T-Junction
ODD per res onto
Figure 16. Ontology for the T-Junction

Example representations:

  • Syntax 1:

    # PERMISSIVE MODE
    permissive_odd: ODD:
        keep(road_topology.properties.lane_dimensions in [3..5]m)
        keep(road_topology.properties.number_of_lanes in [1..2])
    # RESTRICTIVE MODE
    restrictive_odd: ODD:
        keep(road_topology.drivable_area_elements.lane_elements.
             lane_divider.lane_marking == broken)
        keep(road_topology.drivable_area_elements.lane_elements.lanes == single_lane)
        keep(road_topology.properties.driving_direction == left_hand_traffic)
        keep(road_topology.properties.lane_dimensions in [3..5]m)
        keep(all_default_values()) # makes it restrictive
  • Syntax 2:

    # PERMISSIVE MODE (declare a new element to be outside ODD)
    INCLUDE odd_ontology
    MODE Permissive
    SUITABLE laneDimensions is {min: 3.0, max: 5.0}
    SUITABLE numberOfLanes is {min: 1.0, max: 2.0}
    # RESTRICTIVE MODE (declare a new element to be inside ODD)
    INCLUDE odd_ontology
    MODE Restrictive
    SUITABLE Lanemarking is ‘broken‘;
    SUITABLE Lanes is ‘single lanes‘;
    …
    UNSUITABLE driving_direction is ‘left_hand_traffic‘;
    SUITABLE laneDimensions is {min: 3.0, max: 5.0};
Permissive vs. Restrictive ODD

The table below illustrates the permissive and restrictive ODD at both the local (individual statement, i.e. syntax 1: keep(all_default_values()), syntax 2: SUITABLE, UNSUITABLE) level and global (overall permissive and restrictive mode) level. It can be seen that at the individual statement level, the element needs to be specifically listed, by assigning permissive or restrictive qualifier such element can be permitted or restricted. However, at the global level, the specific element does not need to be mentioned in the ODD description itself. The global level permissive or restive mode relies on the underlying ontology to work. If the element is present within the ontology but not listed in the ODD, a global 'permissive' or 'restrictive' mode will auto include or exclude such element. However, if the element is not contained within the ontology and is not mentioned in the ODD description, under 'restrictive' mode it will be excluded even if later on the ontology is extended to include such element. This is due to the fact that 'restrictive' mode restricts everything that is not mentioned in the ODD. On the other hand, under 'permissive' mode, if the element is not in the ontology then it will not be within the ODD either. But if later the ontology gets extended to include such element, then it will be inside ODD. This is due to the fact that under 'permissive' mode, anything that is in the ontology but not mentioned in the ODD will be allowed.

Element specifically listed?

Restrictive

Permissive

Level

yes

Element is outside ODD (e.g, UNSUITABLE T-junction, keep(all_default_values()))

Element is inside ODD (e.g, SUITABLE Motorway), keep(<attribute> == <condition>)

individual statement

No, but element is represented in the ontology

Element is outside ODD

Element is inside ODD

global

No, and element is not represented in the ontology

outside the scope of ODD, even if the ontology is extended

outside the scope of ODD, however, if the ontology is extended the element will be inside ODD)

global

Example:

A given ontology provides the definition of rainfall as low, medium, heavy, and thunder storm.

Weather|_ Rainfall|_ low|_ medium|_ heavy|_ thunder storm
Figure 17. ontology

Restrictive

low rainfall and medium rainfall is allowed

Permissive

cloud burst rainfall is not allowed

cloud burst needs to be defined in the ontology

Weather|_ Rainfall|_ low|_ medium|_ heavy|_ thunder storm|_ cloud burst
Figure 18. updated ontology

4.5. Extensibility

4.5.1. Extension by Function

For the evaluation of ODD boundaries and measures the extensibility by functions is necessary, these function will not be part of the ODD but can be reference in the ODD, to point the evaluation tooling to the correct function. Basic mathematical functions (like sqrt(), max(), min() ) or functions with respect to the application domain (e.g. distance() ) are required for conditions, limits and measures. They can come from supplied library or external binding (e.g. C, Python) For each function the input parameters and the return values must be defined

Example use cases: Bind function

binding function from an external library

Example representations:

  • Syntax 1

    struct ODD:
        def speed_limit():speed  is external.cpp("get_speed_limit", "libutils.so")
        speed: speed
        keep(speed < speed_limit() * 0.95)
  • Syntax 2:

    INCLUDE get_speed_limit, libutils.so
    METRIC speed_limit_out IS speed_limit(get_speed_limit, libutils.so) UNIT 'm/s' TYPE float
    SUITABLE * WHEN speed < speed_limit() *0.95
    
    # METRIC is a potential keyword for defining external functions and libraries, but will need further discussion within the group during the standardisation phase.

Example use cases: Binding a library

binding an external library and use a function from the bound library

Example representations:

  • Syntax 1

    def weatherStatistics(where: list of string, when: list of string, query: string): real\
      is external.cpp("weatherStatisticsQuery", "libDbApi.so") # this binds to external library
    set percentSnowFall: real = weatherStatistics(['Germany'],['2020'], 'frequencyPercent') # call to external library
    set max: real = 1
    def percentageUnsuitableWeather() is (max - PercentSnowfall); # computation using the value returned from the query
  • Syntax 2:

    INCLUDE weather_Statistics, libDbApi.so
    METRIC weather_Statistics IS external.cpp("weatherStatistics", "libDbApi.so")
    MEASURE PercentSnowfall
    	TYPE float UNIT "%"
    	RANGE [min - max]
    	SOURCE ExtDataSource.query(weatherStatistics(['Germany'],['2020'], 'frequencyPercent', weather);
    FLOAT max = 1.0;
    FLOAT min = 1.0;
    METRIC PercentageUnsuitableWeather IS (max - PercentSnowfall)
    
    UNSUITABLE * WHEN (Percent_Snowfall =  max)
    
    # METRIC is a potential keyword for defining external functions and libraries, but will need further discussion within the group during the standardisation phase.

4.5.2. Extensibility of ontology

Due to the diverse stakeholders and use cases, as well as an ever expanding ODD ontology, there are requirements to modify the underlying ontology from a base model. During sharing and exchange of ODD specification, such modifications need to be traceable within the ODD specification. The language therefore shall support the addition of new attributes and values that are not in the original ontology. It shall support the establishment of standard libraries, e.g. PAS 1883.

Example use cases: add an attribute:

Based on the current PAS 1883, user would like to add too_cold attribute defined when the ambient temperature is below 20 degree C. Additionally, user would like to add rain_light attribute defined when rain rate is less than 5 mm/hr and droplet size less than 1mm. too_cold will be added onto the Environment branch of PAS 1883, and rain_light will be added onto the rainfall branch.

Example syntax:

  • Syntax 1:

    # adding another feature to the ODD environment type
    extend ODD_environment_weather_type:
         set too_cold: bool = (temperature < 20Celsius)
    # adding a feature to rainfall type
    extend ODD_rainfall_type:
        light_rain: bool
        keep(light_rain == (rainfall.rainfall_rate < 5mm_per_h and
             rainfall.droplet_size < 1mm))
  • Syntax 2:

    INCLUDE BSI_PAS_1883.odd
    ADDCOND too_cold TO Environment/ODD
    ADDCOND rain_light TO rainfall/weather/ODD
    DETERMINE too_cold WHEN (ambient_temperature < -20)
    DETERMINE rain_light WHEN (rain_rate < 5) AND (droplet_size < 1)
    MEASURE ambient_temperature UNITS C
    MEASURE rain_rate UNITS mm/hr
    MEASURE droplet_size UNITS mm
    
    # Alternatively, the MEASURE could be combined into the DETERMINE statement to be more compact

4.5.3. Metrics

Detailed description of metrics can be found in the Metrics chapter

4.5.4. Probability and uncertainty

This is explained in more detail in the Syntax concepts for uncertainty and probability.

4.6. Integration with Scenario-based Testing Workflow

OpenODD language and format can be viewed as a stand alone language for several use case. In addition, there are many use cases where the format needs to easily be merged and combined with other component , in order to create a complete set of a test case ( or test cases ). When one looks at these test cases, the simulation and/or virtual testing world should be referred to. The next sections will describe several use cases, and the from these the requirements on the format/language will be derived. The key requirement is that that while OpenODD format is defining the ODD boundaries, it has to be integrated with the full testing system that interacts with the environment, and compares it to the ODD.

Several examples of use cases:

  1. The same scenario ( expressed on OpenSCENARIO 2.0) need to be tested with different ODDs. This the scenario code itself, and the test specification need the ability to have a generic reference to an ODD, with assumed taxonomy structure

  2. The scenario and test code need the ability to match current environment conditions to the on specified on the ODD.

  3. The Scenario need to respond to events of environment changes, or ODD conditions changes.

  4. The Scenario need to respond to HMI request and flags, and match them again current ODD definitions and current environment conditions.

  5. The Scenario need the ability to observe actors, and match them against current ODD definitions in place.

Example use cases:

Define a ODD specification constraint using types and enumerations. The ODD specification shall use the same taxonomy as the scenario description.

Example representations:

  • Syntax 1

    import odd_defs.sdl     # the ODD type declarations/taxonomy
    # define odd_1 to be a subtype of any_odd, with extra constraints
    # restricting the permissible conditions
    
    struct odd_1: any_odd:
        keep(road_type in [motorway])
    # odd_1 is not an entity that can be addressed in used in the scenario code
    ODD expressed using OpenSCENARIO 2.0 Enumerated types and constraints: As the ODD taxonomy and definitions are actually expressed in OSC 2.0 language, any scenario code can be easily merged with these definitions. ODD values, struct members and other can be easily referenced and compiled together. Example: The ODD is defined as a struct entity, and can be used anywhere. ODD boundaries constraints are applied to struct members.
  • Syntax 2

    # The example below will corresponding to only the motorway from the odd_defs taxonomy is permitted for the ODD
    INCLUDE odd_defs.sdl
    MODE Restrictive
    SUITABLE Motorway

Compatibility with OpenX standards:

As mentioned above, OpenODD needs to be integrated with the full scenario based testing flow. In order to make this integration as easy and as smooth as possible, the following is expected:

  • Adherence to the same properties and entities in OpenX domain models and ontology.

  • Adherence to the same types, units and expressions syntax and semantics.

  • Adherence to the same conventions for variable names

  • It is also desired that the syntax will be the same – in order to enable re-use and integration

5. Attributes for ASAM OpenODD

5.1. Overview

OpenODD requires a set of standardized "ODD entities" that define and characterize the elements of an ODD and their structural relationships. Several ODD taxonomies already exist in the literature, and some of them have even been standardized to some extent, as BSI PAS 1883 or the AVSC Lexicon.

In the next sections, the following points are discussed: * present a list of requirements for OpenODD attributes * elaborate on current limitations of ODD entities modeling approaches in existing related standards * Propose, motivate and explain the ODD entities modeling approach that openODD takes * Present the list of ODD entities of choice for OpenODD and their relationships, making use of visual diagrams and referencing an interactive HTML file accompanying this concept paper, where the list, structure and metainformation about ODD entities of choice are present.

5.1.1. Requirements for OpenODD Attributes

The general requirements list for ASAM OpenODD includes the set of requirements which are specifically relevant to ODD entities. In addition, it is useful to highlight the following requirements related to ODD entities:

  • The ODD entities need to be hierarchical

  • The ODD entities need to be extensible

  • The ODD entities need to be aligned with other similar standardization activities, where possible

5.1.2. ODD Entities data modeling

Limitations of current ODD entities data modeling approaches

While containing a comprehensive set of relevant ODD entities/attributes, most of the available taxonomies have limitations that do not make them suitable to be used "as is" in a machine readable ODD definition format, such as OpenODD. One of the aforementioned limitations being the lack on a consistent ontological relationships between the ODD entities. For example, while most of the available ODD taxonomies are organized in a hierarchical fashion, often times the nature of the parent-child relationship is not semantically consistent throughout the whole hierarchy, and it is not being formally specified.

In order to cope with the limitations of existing ODD taxonomies, and develop a solution which is compatible with a formal and machine readable representation, we propose the solution detailed in the next section. The solution presented below doesn’t exhaustively capture all elements. Our goal is, instead, to detail a viable approach to model ODD entities and their relationship in a manner that is consistent, extensible and compatible with the other requirements of the OpenODD standard.

OpenODD ODD entities data modeling approach

We propose to organize OpenODD entities in two main taxonomies: ODD Attributes, and ODD Parameters. Each of the taxonomies is modeled (by definition) as a hierarchical tree where the semantics of parent-child relationship means "is a type of". Thus, the more in depth in the tree, the more specialized are the concepts with respect to the parent category. EXAMPLE: For example, we can refer to a simple tree like [Vehicle-→ Car -→ Sports car, Vehicle -→ truck -→ box truck] where every parent category is a superset of the child category, and, ideally, siblings nodes are disjoint sets. The parent-child relationships in our ODD entities are analogous to this example.

The taxonomy of ODD Attributes details out the actual "concepts" that are part of an ODD definition. ODD Attributes are of boolean data type, meaning that either they are, or they are not part of an ODD definition. EXAMPLE: Concepts such as "Environmental conditions", "rain", etc. are modeled in this taxonomy.

ODD Parameters define and quantify ODD Attributes, and are organized in their own tree. ODD Parameters data type may be integer, float, etc. and also include units of measurement (SI units like mm, mm/h, kg, etc), and potentially explicit value ranges. EXAMPLE: Example of ODD Parameters are "ODD Environment Parameter" "Rainfall rate", "droplet size", etc.

Both ODD Attributes and ODD Parameters can have quantifying discretization of child nodes.

ODD parameters are tied to ODD attribute and they define, enrich, augment, and describe the semantics of the attributes.

Please read hierarchy chapter

Linking ODD Attributes to ODD Parameters

OpenODD concept paper presents multiple (at least three) different format and syntax approaches that could be considered viable and fulfill the requirements for the upcoming standardization project.

Different options exist that can provide suitable implementations for defining the relationships between ODD attributes and ODD parameters, either leaning towards a more ontological approach, where such relationships are specified in the underlying ODD ontology, or a more programmatic approach, where such linkages are established through defined functions, classes, or procedures.

While section 4.3.8 of this paper already provides a first overview of syntax and format implementation of the linkage between parameters and attribute, a specific implementation choice will be made in accordance to the syntax and format approach adopted for the overall standard in the subsequent standardization project.

Extensibility of ODD Entities

Similarly to the linking mechanisms between ODD attributes and parameters, the extensibility of the ODD entities list (attributes and parameters) is a matter highly tied to the specific syntax and format of choice. Section 4.4.2 provides an overview of an approach to extend the underlying ODD Ontology with new attributes, parameters, their metrics, units, etc.

5.1.3. OpenODD Standard ODD Entities

OpenODD choice of ODD Entities is well aligned with BSI PAS 1883 standard, as well as the AVS Lexicon //TODO link resource The set of ODD entities proposed for OpenODD concept is the result of an in depth analysis of the aforementioned documents, which takes the entities listed in both, and organizes them in a comprehensive hierarchical structure.

5.2. Requirements for the attributes of ASAM OpenODD

  • Need to be hierarchical

  • Need to be extensible

  • Need to be compliant with existing standards (ASAM OpenX family and non-ASAM standards)

  • Need to be consistent with taxonomies used by ODD taxonomy standards.

  • Need to be parameterizable

5.3. Attributes for ASAM OpenODD

  • the goal is to provide a sample of the possible attributes

Dynamic_Elementsource: BSI PAS 1883Agent_Typesource: BSI PAS 1883Subject_Vehiclesource: BSI PAS 1883ODD_TaxonomyEnvironment_Elementsource: BSI PAS 1883Connectivitysource: BSI PAS 1883Illuminationsource: BSI PAS 1883ParticulatesWeathersource: BSI PAS 1883SubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOf
Attribute: Agent_Type
Agent_Typesource: BSI PAS 1883Special_Vehiclesource: BSI PAS 1883SubclassOf
Attribute: Connectivity
Connectivitysource: BSI PAS 1883V2Vsource: BSI PAS 1883V2Isource: BSI PAS 1883V2PFleet_Managementsource: AVSC LexiconObstructionsource: AVSC LexiconPositioningsource: BSI PAS 1883Galileosource: BSI PAS 1883GLONASSsource: BSI PAS 1883GPSsource: BSI PAS 1883V2Xsource: BSI PAS 1883Cellularsource: BSI PAS 1883Satellitesource: BSI PAS 1883WiFisource: BSI PAS 1883SubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOf
Attribute: Illumination
Illuminationsource: BSI PAS 1883Artificial_Illuminationsource: BSI PAS 1883Natural_IlluminationSubclassOfSubclassOf
Attribute: Particulates
ParticulatesDrizzlesource: AVSC LexiconHazesource: AVSC LexiconMarine_Particulatessource: AVSC LexiconNon_Precipitating_Dropletssoruce: BSI PAS 1883Fogsource: AVSC LexiconMistsource: AVSC LexiconSand_and_Dustsource: BSI PAS 1883Smoke_and_Pollutionsource: BSI PAS 1883Volcanic_Ashsource: BSI PAS 1883SubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOf
Attribute: Weather
Weathersource: BSI PAS 1883Precipitationsource: AVSC LexiconHailsource: AVSC LexiconRainfallsource: BSI PAS 1883Freezing_Rainsource: AVSC LexiconSleetsource: AVSC LexiconSnowfallBSI PAs 1883WindBSI PAs 1883SubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOf

5.3.6. Parameters for ASAM OpenODD

ODD_ParameterEnvironment_Element_Parameterwind_speedsource: BSI PAS 1883type: floatunit: m/sRainfall_Ratesource: BSI PAS 1883type: floatUnit: mm/hLight_Rainfall_Ratesource: AVSC Lexiconrange: 0 < 2.5Moderate_Rainfall_Ratesource: AVSC Lexiconrange: 2.5 - 7.6Heavy_rainfall_Ratesource: AVSC Lexiconrange > 7.6Sun_Elevationsource: BSI PAS 1883type: floatunits: Degrees/radiantsIlluminancesource: AVSC Lexicontype: floatunits: LuxVolume_of_Trafficsource: BSI PAS 1883Density_of_Agentssource: BSI PAS 1883Flow_Ratesource: BSI PAS 1883Dynamic_Element_ParameterSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOfSubclassOf

6. Metrics

6.1. Understanding metrics

At first glance, ODD seem to be a merely qualitative list of objects and conditions that can exist in the specified working environment of an automated vehicle. However, the variety of use cases for ODD specification listed in this document suggests that in several project phases engineers want answers on many specific aspects of ODDs, or at least related to ODDs. These can include pure Boolean answers (e.g. Is a specific scene of a driving scenario inside vs. outside the ODD), answers on a qualitative scale (e.g. Is the present coverage of an ODD by a set of test scenarios poor, medium or good) or even quantitative answers (e.g. Coverage of equivalence classes of an ODD in percent). Just listing attributes and equivalence classes that define the ODD would not be sufficient in a safety and quality-oriented automotive development and release process. Therefore, the OpenODD language will have to provide means to define certain metrics (which does not mean that the OpenODD standard comes with an exhaustive list of such metrics; it is rather about enabling the language users to define them).

A definition of metrics for the purpose of the OpenODD language will be given later down. As an intuitive introduction, a metric is a function that maps some property about a thing (some quality attribute, for instance) or about several related things (some sort of a distance between them, for instance) onto an ordered scale (for instance the set of non-negative real numbers, or a point scheme reaching from 1 to 10 points). Metrics allow propositions about the quality, completeness, work progress etc. related to ODD specification or ODD usage (e.g. to derive vehicle requirements, to specify test or simulation scenarios). In some cases they allow comparisons between automated driving system with different ODDs (which AD feature allows the "bigger" ODD?) or between different test scenario sets aiming at covering a given ODD. It should be noted that ODD related metrics do not necessarily deal only with the ODD; they can involve propositions about a scenario or a set of scenarios, about performed test runs on a simulator or public road, about currently implemented vehicle features with respect to certain ODD conditions and many more.

Metrics can be defined on objects or situations, in particular on their various attributes by using logical or arithmetical operators (like AND, OR, +, -, greater than, less than) or mathematical functions (e.g. min(), max(), log() etc.). Some involve domain-specific functions (e.g. distance(), maxspeed() etc.) or queries to an ODD or scenario database for objects or conditions matching certain filters (e.g. select all roads in a given country with at least four lanes). Complex metrics can be calculated from other metrics that have been defined before.

6.2. Overview

This chapter elaborates requirements not only to the OpenODD language in general, but more specifically also to the language elements allowing the definition of attributes and equivalence classes of an ODD. The requirement to the language in general will be that it is able to host metrics definition and metrics usage as optional data elements associated with an ODD or objects or attributes thereof.

To host means that the grammar of the language has keywords (e.g. "define metric") and operators, e.g. mathematical operators, to define metrics and also to refer to existing metrics definition, i.e. to use them. A potential usage is to refer to an existing metric to define a new metric. The metric definitions made by the user can be stored in the same or an included OpenODD file. But we don’t mean by "hosting" that the OpenODD language does come with built-in metrics; metrics are in the language user’s responsibility. It can be decided whether or not the OpenODD language utilities will be shipped with some sort of a standard library that defines already some frequently used metrics

Right below, a definition of metrics specifically for the purpose of ASAM OpenODD is given. Next, a couple of user stories for different roles in the development and validation process of automated driving functions are listed to better understand the use cases and requirements for OpenODD metrics. Note that the formal requirements, also with regards to metrics, are included in the global OpenODD requirements list. Finally, some examples are given how metrics could be calculated and how their definition and usage might look in different candidate syntaxes for the OpenODD language. In the stage of this concept paper we do not attempt to give a fully proven mathematical foundation, nor a final syntax for metrics.

6.2.1. Definition of Metrics in the Context of ASAM OpenODD

A metric is a function that transforms features or properties of an ODD - alone or in its relation to other ODDs, to scenarios etc. - into a number or an ordered set of qualifiers (e.g. good > medium > bad). It is not required that the function is mathematically defined and objective. Sources for a metric can also be expert judgement, as long as the metric is clearly declared in terms that the syntax is accessible to interpretation by a machine (e.g. in order to perform some comparison or statistics on it) and sufficiently reproducible in practice. However in most practical cases the metrics calculations will be based on a computing language, using machine readable data sources.

An example for the latter would be the severity ratings S0-S3 as defined in ISO 26262-3: They are not mathematically defined, but clearly described in terms of "injuries of AIS classification X are at least Y% probable". Experienced accident researchers usually come to very similar results for the same setting. However, the latter relaxation was just introduced in order not to exclude any human-centered or semi-quantitative metrics in cases where we don’t have any better option today - the main focus will be on metrics that can objectively computed from an ODD definition (and related formal descriptions, e.g. of scenarios).

Notes:

  • The definition and interpretation of these numbers and qualifiers is user-specific and not necessarily accessible to an objective formal decision (e.g. „storm“ as the highest level of wind as opposed to „wind speed > 100 km/h“)

  • The underlying measures can be objectively physically measurable quantities, like lane width in m or rain intensity in mm/h. In the same way, physically objective ODD metrics can be formed, e.g. distance to leaving ODD in m.

  • By using metrics engineers can compare two ODDs, e.g. a system built for ODD2 can operate on 20% more German roads (or road kilometers) than a system built for ODD1, depending on how frequently certain ODD features statistically occur on German roads, or per road kilometer.

  • Metrics can relate scenarios, sets of scenarios or recorded road test data to an ODD, e.g. "What percentage (of km driven, or of time driven) of Scenario X is inside ODD Y?" or " What is the coverage of Scenario Set X with respect to ODD Y?".

  • ODD metrics can serve as foundation for metrics on scenarios and the quality of a Vehicle-Under-Test (VUT).
    E.g. „In how many cases did the VUT correctly perceive the edge of an ODD before leaving of the ODD occurred?“

6.3. User stories for metrics of ASAM OpenODD

In this chapter the user stories for metrics are listed, these are based on the user stories listed in the beginning of this document. Using the mentioned user stories in this chapter regarding metrics, it is possible to derive the requirements for metrics. The user stories for metrics may differ from the user stories defined in Use cases for ASAM OpenODD, this is caused by the fact that metrics are an application using the ODD specification, while the user stories in Use cases for ASAM OpenODD, are focused on the format and language of OpenODD. It is important that the requirements from are metric side are known, when developing OpenODD, as the format / language needs to consider the use of metrics with the ODD specification.

6.3.1. Specification and/or quality engineer

Representativeness of ODD specification in comparison to real driving environment

As an ODD specification engineer, I want to measure the representativeness of my ODD specification to a real driving environment.

Example

Approach

Percentage of aspects allowed (or not explicitly forbidden) by an ODD specification w.r.t. a given ontology

Use algorithms like calculation of an aspect-wise norm and IoU to determine the overlap of the addressed (i.e. explicitly allowed or explicitly forbidden) attributes in the ODD specification with all available dimensions and value ranges in the ODD.

Percentage of aspects addressed by an ODD specification w.r.t. a (preprocessed / labeled) representation of relevant attributes in a set of driven scenarios or a statistical evaluation of traffic scenarios.

Use algorithms like calculation of an aspect-wise norm and IoU to determine the overlap of the addressed (i.e. explicitly allowed or explicitly forbidden) attributes in the ODD specification with all available dimensions and value ranges in the ODD.

Percentage of aspects allowed (or not explicitly forbidden) by an ODD specification w.r.t. a description of the requirements for a given target country or market (e.g. which road types exist in Germany, etc.)

Use algorithms like calculation of an aspect-wise norm and IoU to determine the overlap of the addressed (i.e. explicitly allowed or explicitly forbidden) attributes in the ODD specification with data about real road traffic and environmental conditions, which can be retrieved from databases with public traffic statistics or preprocessed and labeled data from test rides. Optionally the metric can be weighted by the relative statistical importance of individual dimensions or values.

Coverage of discrete values or value ranges of relevant attributes in a set of driven scenarios or a statistical evaluation of traffic scenarios by a given ODD.

Use algorithms like IoU to calculate the overlap of a set of driven scenarios with the ODD specification and compare the actual occurrence of situations with the ODD specification.

Statistical similarity of the probabilities assigned to single or combined values of different aspects with the corresponding statistical data from public sources or test rides (e.g. sum of squared errors).

Preconditions: ODD specification is enriched with probability values for the situations or attribute values. Calculate a quality factor (e.g. sum of squared errors) that summarizes for all dimensions (single or combined) the statistical probabilities given by the ODD specification with collected data from test rides or public statistical material..

Quality and completeness of ODD specification

As an ODD specification engineer or a quality engineer, I want to measure the quality and completeness of my ODD specification to see what needs to be improved or when I‘m done.

Example

Approach

Absolute number of attributes addressed by the ODD specification.

Sum all attributes that are mentioned (explicitly allowed or explicitly forbidden) by the ODD specification.

Ratio of unaddressed (neither explicitly allowed or forbidden) attributes or attribute values / value ranges in an ODD specification and the total available set of values / value ranges defined by the underlying ontology or a golden sample ODD specification or a (preprocessed / labeled) representation of relevant attributes in a set of driven scenarios that characterize the target market.

Number of all attributes addressed (i.e. explicitly allowed or explicitly forbidden) or unaddressed (neither allowed nor forbidden) in the ODD specification, divided by total number of attributes from reference source. Attributes can also be weighted by importance for the market acceptance etc."

As the language is OpenODD by assumption, many quality attributes are fixed and don’t need to be measured (e.g. degree of formality, understandability etc.) - so completeness is the most relevant remaining quality property.
Size of the ODD (i.e. how many environments are inside the ODD)

As an ADAS/AD designer, I want to measure the size and restrictiveness of my ODD or compare several ODDs by size or details to see how broad the usage scope of my ADAS is and how sure I can be about remaining risks.

Example

Approach

(Weighted) attribute values or value ranges that are forbidden (or not explicitly allowed, depending on specification style) by an ODD spec w.r.t. the ranges defined by the underlying ontology, or by an evaluation of actual driving scenarios, or by the regulations or situations in a given target country.

+ Particular example: What percentage of all road kilometers in Germany are suitable for a given automation feature according to its ODD specification (generally, or related to specific dimensions like road type, weather etc.)?

Sum all attribute values (e.g. for road type: highway and country road) that are allowed by the ODD specification and divide them by all values that the ontology provides for that attribute (e.g. for road type: highway, country road, city street, off road). Optionally give them a weight according to their market importance (e.g. if the chances of snow in some target country are very low, give snow a lower weight in comparison to sunshine and rain). In a modified version this can also be applied to attributes with continuous value ranges (like lane width or visibility range).

Comparison of two given ODD specifications by size (how much bigger is ODD one ODD in comparison to the other)?"

Check wether one ODD1 is a subset of ODD2. Divide the norm (attribute-wise or aggregated to one weighted sum) of the smaller ODD by the norm of the bigger ODD."

6.3.2. Data scientist

Coverage by the ODD or some ODD aspects by a scenario library

As a data scientist, given a scenario library and optionally a distribution of parameter values to concretize these scenarios, I need to provide a coverage metrics for the safety case.

Example

Approach

Coverage of discrete values or value ranges of relevant attributes in a set of driven scenarios or a statistical evaluation of traffic scenarios by a given ODD.

Use algorithms like IoU to calculate the overlap of a set of driven scenarios with the ODD specification and compare the actual occurrence of situations with the ODD specification. Option A: Apply equivalence partitioning to all dimensions of the ODD that are not discrete by nature. All combinations of such equivalence classes form bins, of which the occurrences are collected. Monitor during scenario execution which parts of the scenario fall into which equivalence classes and count the occurrences for each bins. Define a minimum coverage (e.g. at least 10 occurrences per bin) and count for which percentage of bins the objective is already met by the set of scenarios. Option B: Assign to scenarios (which is an "infinitely thin line", not a space) a certain "width", i.e. a range of neighbor values per category that are considered as covered. E.g. driving with a speed of 50 kph could be considered as representative for every speed in the range 40..60 kph, likewise for weather conditions etc.. So every executed scenario is now a sub space of the ODD (or even the space beyond ODD limits). Form the union set of all executed scenarios, then calculate the IoU with the ODD.

Exposure to scenarios or specific aspects of the ODD

As a data scientist, given an ODD, I need to determine the exposure to a specific scenario or to certain aspects of a scenario (e.g. certain weather conditions) specified in the ODD specification, so that I can determine the weight/contribution of the scenario results in the overall assessment.

Note: The term "exposure" can for instance be understood as "exposure" in ISO 26262-3 Annex B.3 in combination with the Note 1 to the definition of "exposure" in ISO 26262-1: "The parameter 'E' in hazard analysis and risk assessment represents the potential exposure to the operational situation." Example: "Medium Exposure Probability (E3)" corresponds to 1%…​10% of average operation time of the vehicle.

Example

Approach

Exposure (relative portion in comparison to all driven hours or km) of values (discrete case) or value ranges (continuous case) per dimension

Retrieve statistics of conditions in target country or aggregated and labeled results from many test rides and put into bins per value or equivalence class for each dimension of the ODD. Divide number of cases per total number of cases.

Exposure for a combination of values (e.g. night and rain and highway and low traffic density) or a given scenario"

value table or probability distribution -→ n/a for now (Metric in a broader sense)"

6.3.3. Scenario Editor

Inclusion of a scenario within the ODD

As a scenario editor, given an ODD, I need to determine if a scenario lies entirely within the ODD.

Note: This is, at first, not a metric but a query, which is further explained in the section on ODD queries. But there are subsequent metrics, of which a few examples are given here.

Example

Approach

What percentage (of km, of time) of a scenario lies within the ODD?

Form the intersection of the scenario and the ODD (note that one has the appearance of a line whereas the other has the appearance of an area or a space, so additional considerations like giving a width to a scenario by making a shift from concrete to logical scenarios, or partitioning the ontology space are necessary in preparation). Divide the norm of the cut set by the norm of the entire scenario.

What percentage of a scenario lies within the ODD with regard to a specific condition (e.g. lane width)?

Same as 3., but decide the in/out question only by one specific dimension (attribute) of the ODD, e.g. the weather

For how long (in time or in distance) did the ODD violation of a given scenario persist (generally, or with regard to specific attributes)?"

Use intersection to filter out the parts of a scenario that lie outside the ODD and measure the time or distance driven of these scenario fragments, form maximum if needed."

As a scenario editor, given an ODD, I need to determine if

(a) a scenario induces a violation of ODD attribute boundaries,

(b) which attribute boundaries of the ODD may be violated by the scenario,

(c) by how much or

(d) how long they may be violated, which are important information when judging the fall-back or Minimal Risk Manoeuvre capability of an ADS.

Note: Cases (a) and (b) are queries (providing Boolean answer or selection or attributes or values for an attribute) and therefore discussed in the section on queries. Cases (c) and (d) are metrics (providing a real number or similar as an answer) and therefore discussed in the section on metrics.

Example

Approach

(c) Max value in centimeters by which the minimum lane width specified by the ODD spec was ever undercut.

(c) For every part of the scenario that lies outside of the ODD because of a specified attribute (e.g. lane width) find the extreme value (i.e. min or max, whichever is further apart from the ODD definition boundary) and return this value in its natural unit (e.g. centimeters for lane width, mm per hour for rain strength etc.) - This metric works only for attributes with a continuous value range

(d) Time in seconds for which we have been operating (or will be operating, if the scenario is just under definition) outside of the ODD (in general or related to one selected ODD attribute, like lane width)"

(d) Determine from a recorded run or from a projection of the scenario under planning the point of time when the vehicle is leaving the ODD (in general or related to a selected attribute) and from there count the time until re-entrance into the ODD or end of the scenario or occurrence of a specified event (e.g. triggering of an emergency reaction by the vehicle under test)"

6.3.4. Safety Engineer

Coverage of the ODD by a situation catalog for Hazard Analysis (Functional Safety, SOTIF)

As a safety (SOTIF) engineer, given an ODD and a Hazard Analysis (acc. ISO 26262 / ISO 21448), I need to determine the coverage of that situation set w.r.t. the ODD, or

Example

Approach

Weighted percentage of discrete values or value ranges of relevant attributes in the HARA situation set with respect to the same attributes in the ODD.

Use algorithms like calculation of an aspect-wise norm and IoU to determine the overlap of the addressed (i.e. explicitly allowed or explicitly forbidden) attributes in the ODD specification with the list of situation attributes and the combinations thereof. Optionally the metric can be weighted by the relative statistical importance (exposure) of individual dimensions or values. It should also be checked that the situation catalog entries are actually used in the HARA.

Exposure (occurrence probability) of scenarios or specific aspects of scenarios in the ODD

As a safety (SOTIF) engineer, given an ODD and a Hazard Analysis (acc. ISO 26262 / ISO 21448), I need to determine the exposure ("E-Factor") to certain situations (e.g. highway is 10% of average driving duration, emergency braking happens not more than 5 times per year on average etc.) based on the given ODD, so that I can classify the risk of a hazardous behavior in these situations during Hazard Analysis and subsequent SOTIF activities.

Note: The term "exposure" can for instance be understood as "exposure" in ISO 26262-3 Annex B.3 in combination with the Note 1 to the definition of "exposure" in ISO 26262-1: "The parameter 'E' in hazard analysis and risk assessment represents the potential exposure to the operational situation." Example: "Medium Exposure Probability (E3)" corresponds to 1%…​10% of average operation time of the vehicle.

Example

Approach

Exposure (relative portion in comparison to all driven hours or km) of values (discrete case) or value ranges (continuous case) per dimension

Retrieve statistics of conditions in target country or aggregated and labeled results from many test rides and put into bins per value or equivalence class for each dimension of the ODD. Divide number of cases per total number of cases. Optionally translate the percentages into the ISO 26262-3 scale (e.g. E4 = more than 10%, E3 = 1…​10% etc.)

Exposure for a combination of values (e.g. night and rain and highway and low traffic density) or a given scenario"

value table or probability distribution → n/a for now (Metric in a broader sense) which allows calculation of combined situations (e.g. at night and at rain and on a highway). Note that in the general case the exposures to the individual dimensions are not stochastically independent.

Coverage of the ODD by test scenarios as a means to demonstrate sufficient level of safety

As a safety (SOTIF) engineer, given a scenario library and an ODD, I need to determine suitable metrics for coverage of the ODD by validation scenarios, so that I can assess the progress of my safety validation and estimate the remaining risk due to unknown hazardous scenarios that still lie inside the boundaries of my ODD.

Example

Approach

(Weighted) Coverage Metrics for a Scenario Set vs. an ODD

Form the intersection of the union set of all scenarios and the ODD. Note that a scenario has the appearance of a line whereas the ODD has the appearance of an area or a space, so additional considerations are necessary, as explained below. Option A: Apply equivalence partitioning to all dimensions of the ODD that are not discrete by nature. All combinations of such equivalence classes form bins, of which the occurrences are collected. Monitor during scenario execution which parts of the scenario fall into which equivalence classes and count the occurrences for each bins. Define a minimum coverage (e.g. at least 10 occurrences per bin) and count for which percentage of bins the objective is already met by the set of scenarios. Optionally apply weighting to the bins, e.g. directed by their exposure / occurrence probability. Option B: Assign to scenarios (which is an infinitely thin line, not a space) a certain width, i.e. a range of neighbor values per category that are also considered as covered by the scenario. E.g. driving with a speed of 50 kph could be considered as representative for every speed in the range 40..60 kph, likewise for weather conditions etc.. So every executed scenario is now a sub space of the ODD (or even the space beyond ODD limits). Form the union set of all scenarios, then calculate the IoU with the ODD."

6.3.5. Development engineer

Performance test of runtime decision making about inside/outside ODD

As a development engineer I want to assess or test the performance of runtime decision algorithms that tell whether I’m about to leave the ODD, and to obtain a probabilistic estimate for the risk of a wrong decision.

Example

Approach

  1. False positive rate (system reports inside ODD when actually outside

  2. False negative rate (system reports outside ODD when actually inside

For 1.) and 2.): Record ground truth (inside or outside ODD) and output of the onboard decision algorithm over time for many test rides or simulation runs. Takes samples (e.g. at equidistant time intervals, or 5 seconds after entering or leaving the ODD) and divide number of inside ODD (or outside ODD) answers by number of ground truth inside/outside ODD samples.

Average / maximum detection time after leaving the ODD"

Drive or simulate scenarios that contain a situation where the ODD is left. Record ground truth time instance for leaving the ODD and change in the output of the onboard decision algorithm and sample average time interval in between (special treatment for cases where leaving the ODD was not reported at all.)"

Testing against the ODD specification

As a development engineer I want to check tests, scenarios, recordings, etc. against the ODD specification.

Example

Approach

How well does a given set of scenarios cover the ODD? (May need equivalence class partitioning and/or weighting)?

Option A: Apply equivalence partitioning to all dimensions of the ODD that are not discrete by nature. All combinations of such equivalence classes form bins, of which the occurrences are collected. Monitor during scenario execution which parts of the scenario fall into which equivalence classes and count the occurrences for each bins. Define a minimum coverage (e.g. at least 10 occurrences per bin) and count for which percentage of bins the objective is already met by the set of scenarios.

6.4. Concepts

Metrics on ODDs are an advanced topic (and therefore not part of the core language, but an extension). The definition and calculation of metrics makes use of many other (extension) concepts of the OpenODD language, and also the underlying ontology (e.g. defined in OpenXOntology language). In particular, concept of the following extension packages may be useful:

  • Math functions: As metrics produce numbers, there may obviously be a need to use mathematical functions that come from libraries (and that are provided by existing functions of a tool chain or written in a general purpose programming language like Python). Examples include sum(), average(), max(), min() or even more advanced statistical functions.

  • Data access functions: Many pieces of information needed to calculate a metric relate to the ODD, but are not part of it. For instance, if the question is what fraction of all usage times are suitable for a given AD function based on its requirements on visibility, then it will be necessary to consult an external database or file containing such information. This can be achieved by adapter functions that also come from extension libraries.

  • Partitioning: If not yet done on ontology level or when defining the ODD itself in OpenODD language, metrics may require some partitioning. For instance, if lane width is defined as a real number in meters, a metric may still what to calculate the fraction of narrow roads within the ODD, so a definition of narrow, medium and wide roads must be created. This can be done using the Partitioning and Equivalence Classes extension.

  • Exposure: Equivalence classes within the ODD can optionally be decorated with an exposure attribute (how frequently is this subclass of the ODD occurring on average driving, which is an important input to hazard analysis and risk assessment). Some metrics have for purpose to calculate such exposures from external statistics or scenario recordings, and use concepts from Exposure extension package.

  • Queries: It may be necessary to retrieve items of information on the ODD or external information sources like road maps, e.g. to find all roads of a certain road type.

Based on these concepts, new metrics can be declared by a given OpenODD syntax and keyword and can be defined using standard mathematical operators (+,-,\*,/) and the aforementioned mathematical functions like min(), max(), average() etc.

Detailed concepts how to implement the typical metrics examples from previous subsection will be elaborated in the standardization phase.

Some promising approaches examined so far are based on the interpretation of an ODD as a multi-dimensional vector of sets:

Imagine an ODD as a multi-dimensional set: E.g. if you have and underlying ontology defined by three enumerative (i.e. mappable to the set of natural numbers N) parameters, for instance

  • road type in terms of {highway, country road, city street}

  • the number of lanes (as a natural number)

  • weather in terms of {sun, rain, snow},

and two more parameters described by continuous values (mappable to the set of real numbers R), for instance

  • visibility distance in m and

  • lane width in m,

then any ODD on this ontology is defined as a subset of N x N x N x R x R, or if you want, as a 5-dimensional vector of subsets (in case of enums) or sub-ranges (in case of continuous values) of the individual sets that form the dimensions of the ontology.

A measure for the size of a subset for each dimension can be determined, e.g. the cardinality (number of elements) for enumerative attributes or the sub range as a fraction of the total range (e.g. ontology allows lane width between 2 m and 4 m, the ODD comprises anything greater or equal than 2.5 m, so the range within ODD is 75% of the total possible range for the dimension lane width).

Set operations like intersection or union can be formed on the sets that define each dimension of two given ODDs.

Combining set operations and size measures can render numerical weights for each dimension of the ODD.

If a single weighted number for the ODD as a whole (not just for each dimension separately) is required as a metric, weighting can be applied by component-wise multiplication with a weight-vector to provide a new, weighted vector.

If a ratio between measures is required, component-wise division can be applied to obtain a vector with numbers between 0 and 1.0 for each component. Again, weighting can be applied to consolidate these numbers to one single measure.

To compare two ODDs (and for many more purposes), the concept of the Jaccard Index or Intersection-over-Union (IoU) (see https://en.wikipedia.org/wiki/Jaccard_index), which is a similarity measure commonly used in object perception and other domains, can be adapted to ODDs.

A sketch of such an IoU adaptation could look like this:

similarityMetricVector(ODD1, ODD2) = componentWiseDiv(weightODDVector(myWeightsVector, intersect(ODD1, ODD2)), weightODDVector(myWeightsVector, union(ODD1, ODD2));

similarityMetric(ODD1, ODD2) = scalarProduct(componentWeightsVector, similarityMetricVector);

6.4.1. Some examples how ODD metrics could look like

The actual syntax for metrics and for the OpenODD language in general will only be defined as part of the standardization project, so it is not possible to provide working and syntactically valid examples at this stage. However, since the topic of metrics is quite abstract for many readers, some examples are given in a pseudo code style that might look similar in the final OpenODD language, but without anticipating design decisions about the final syntax at a later stage.

An example based on an ODD and external weather statistics

Setting: We want to find out what percentage of situations in some country (specifically related to the weather restrictions from ODD specification) are suitable to switch on automated mode for some vehicle. The ODD allows any weather but snowfall. We assume that a weather statistics database for selected countries and selected calendar years in the past is available and can be accessed via OpenODD extension library (we consider different programming languages for choice to implement this access function).

  • Sample Syntax 1

        def weatherStatistics(where: list of string, when: list of string, query: string): real\
            is external.cpp("wheatherStatisticsQuery", "libDbApi.so")
        set percentSnowFall: real = weatherStatistics(['Germany'],['2020'], 'frequencyPercent')
        set max: real = 1
        def precentageUnsuitableWeather() is (max - PercentSnowfall);
  • Sample Syntax 2:

    INCLUDE Math; 		# OpenODD Standard Lib, can also be a python lib
    INCLUDE Statistics;	# OpenODD Standard Lib, can also be a python lib
    INCLUDE Units;		# OpenODD Standard Lib, can also be a python lib
    
    INCLUDE Metrics; 	# OpenODD Extension Package that defines base class for metrics
    INCLUDE Queries; 	# OpenODD Extension Package that defines base class for queries on an ODD definition
    INCLUDE ExtDataSource;	# OpenODD extension to access
    
    INCLUDE MyOntology; 	# This enables the use of all attributes within "MyOntology" incl. weather and roadtype
    
    SUITABLE [roadtype.highway roadtype.urban_roads roadtype.parking_lots] EXCEPT WHEN weather.snowfall;
    
    
    # Define a Metric: Percentage of time where the usage of my AD function is allowed in terms of weather conditions
    
    # Define reusable variables for min an max values.
    FLOAT max = 1.0;
    FLOAT min = 1.0;
    
    # Statistical Percentage where AD function cannot operate due to weather conditions
    MEASURE PercentSnowfall
    	TYPE float UNIT "%"
    	RANGE [min - max]
    	SOURCE ExtDataSource.query(weatherStatistics(['Germany'],['2020'], 'frequencyPercent', weather);
    
    
    METRIC PercentageUnsuitableWeather IS (max - PercentSnowfall);

7. Uncertainty representation in ASAM OpenODD

7.1. Overview

To ensure that an autonomous vehicle is safe to both passengers and other traffic users, many regulatory environment require that 1) manufacturer provide a description of the ODD for this the vehicle is designed to operate, and 2) operators ensure to restrict operation to the ODD.

For details on what ODDs are and how they are defined, please refer to Understanding ODDs

7.1.1. Motivation

Excluding fuzzy and uncertain situations from the ODD restricts the availability of the autonomous vehicle. It is not practical to categorically exclude all situation in which a non-zero risk exists for unsafe outcomes. As an example, it is not practical to exclude freeways even though we know there is a low, but non-zero chance, to encounter people on the freeway, either due to accidents, or due to traffic jams, or some other reasons.

An ODD specification needs to determine in which situations is the vehicle required to drive safely. In other words, the ODD needs to decide, for each situation, whether it is excluded or not. The uncertainty working group is focused on making a black-and-white in-or-out determination when situations are fuzzy or otherwise are associated with uncertainty.

In the concept project the working group targeting the representation of uncertainty within the ASAM OpenODD language, established a list of requirements. These requirements do not contradict concept and requirements staten in the requirement clusters in Requirements for ASAM OpenODD. The requirements agreed upon are as follows:

  • Enable safety argumentation, acceptable risk analysis and HARA against ODD definition

  • Introduce statistical measures, chances of collisions, etc from available data.

  • It is complementary to safety and risk relevant standards

  • Support both quantitative and qualitative exposure specification, but exclude (for now) severity and controllability

  • Support definition of perception uncertainty requirements, sensor capability/limitations

  • Support definition acceptable risk against the ODD

  • Support definition of expected behavior of other road users, risk that pedestrians on motor way

  • Support references to external accident databases

  • Support various validation methods

7.2. Requirements for the uncertainty representation of ASAM OpenODD

  • The ODD must map every situation to an in or out of ODD binary classification

  • Represent Risk of Occurrences

    • Support discrete occurrences, for example pedestrians on the road

    • Support continuous occurrences, for example a stretch of icy road surface

  • Allow extending to represent other types of “newly discovered” uncertainties

  • Allow specification at multiple levels of details (LOD):

    • Level 0: No details, no uncertainty (baseline)

    • Level 1: Uncertainty of weather (need external data sources)

    • Level 2: Uncertainty of occurrence (need external data sources)

    • Level 3: …​

  • Propositional (true/false) class attributes representing which is true in the worlds must be separate from measures.

  • Propositional (true/false) class attributes may be associated with qualifiers which are distinct from all other attributes and measures.

  • Allow ODD reference data originating from external data source.

    • Reference external data associated with weather occurrences

    • Reference external data associated with occurrences associated with traffic patterns and behaviors

7.3. Concepts

The following concepts need to be integrated as optional constructs using the extension mechanism defined in Extensibility.

The concepts in this chapter are explained using the example syntax2 and is explained in Introduction to syntax 2. Other syntaxes may use different keywords or constructs to illustrate the same concept.

7.3.1. Uncertainty Level of Detail

Statements in the language may be qualified or not. Non qualified statements are of lower level of detail than the qualified statements.

Data points necessary to determine if the situation/scene/scenario is within the ODD or not:

baseline statement
SUITABLE Freeway EXCEPT WHEN Pedestrian
increased level of detail
SUITABLE Freeway EXCEPT WHEN Pedestrian@P8

The level of detail increases as we increase the number of qualified statements within the ODD specification.

The source of uncertainty information is outside of the ADS. The on board systems are not required to estimate the uncertainty. As such pedestrian@Q8 will not change its truth value frequently thus the ADS will not exit and re-enter the ODD due to sporadic detection of pedestrians. Therefore using qualified statements avoid flickering of the ODD status.

ULOD Flickering

ULOD ODD Flicker

This enables different stakeholder to leverage different levels of detail of the ODD. For example, the regulators may use the lower level ODD (non-qualified) whereas the manufacturer may use a higher level of detail (qualified).

7.3.2. Uncertainty Qualifiers Approach

Qualifiers represent qualitative probability in terms of upper bound on the occurrence rate. As an example, P8 is a qualifier. The proposition Pedestrians@P8 evaluates to "true" if the rate of pedestrian occurrence is lower than once in 10^8 hours. That occurrence rate is provided by sources external to the driving system. It is typically associated with a time frame, or a geographic region, or both. It is expected that this rate does not change frequently and is associated with relatively wide geographic regions.

Exposure can be bounded by qualifiers, because qualifiers represent an upper-bound of the probability of occurrence. For example, restricting the ODD to Pedestrians@P8="false" (e.g using SUITABLE * EXCEPT WHEN Pedestrians@P8) implies that the ADS is allowed to operate either when pedestrians are not present (=probability of zero) or they are present with a rate lower than once in 10^8 hours. The bound "present with a rate lower than once in 10^8 hours" quantifies and controls the exposure.

Qualifiers are used to achieve True/False mapping of fuzzy uncertain situations, regardless of the root-cause of "fuzzyness" or uncertainty. Each proposition may optionally be qualified by multiple qualifiers.

A qualifier attaches the uncertainty or exposure value to the ODD attribute.

condition of a condition including a qualifier
SUITABLE <condition> EXCEPT WHEN <condition>@<qualifier>

The ODD specification without qualifiers will be call "baseline ODD" and with qualifiers will be called "qualified ODD".

Example Syntax of Qualifiers

Qualifiers may be added to any proposition (not to a metric) as a comma separated list after @ symbol. For example, a qualifier called "P8" can be defined to evaluate to True only when the probability of pedestrians if <10^-8:

suitable condition with qualifiers of probability
SUITABLE Freeway EXCEPT WHEN Pedestrians@P8

Multiple qualifiers can be added as well. Consider a qualifier C2 used to indicator that the confidence level in pedestrian detection is >99%. Both P8 and C2 can be specified as follows

suitable condition with qualifiers of probability and confidence
SUITABLE Freeway EXCEPT WHEN Pedestrians@P8,C2

variable definition statement for qualifiers are as follows:

DETERMINE_QUALIFIER P8 WHEN occurrence_probability <10^-8" whereby the metric 'occurrence_probability' is a reserved keyword of which meaning is defined in the standard.

DETERMINE_QUALIFIER P8 MORE STRINGENT THAN P7 to specify relationship between qualifiers (see semantics below)

Semantics of Qualifiers

The qualifiers are relaxing the conditions accepting situations in which the main proposition would reject; they are negating the main proposition.

Without adding @P8, the above statements maps a situation to True if and only if Freeway=True and Pedestrians=False. With the addition of @P8, the above statements maps a situation to True if and only if freeway=True and (pedestrians=False or P8=True).

Adding multiple qualifiers accepts more situations. For example, without qualifiers, the statement "SUITABLE Freeway EXCEPT WHEN Pedestrians" accepts situation if and only if Freeway=True and Pedestrians=False. Adding @P8,C2 would accept situations if and only if freeway=True and (pedestrians=False or P8=True or C2=True).

Intuitively this makes sense:

  1. When the probability of pedestrians is very low, the risk is acceptable, thus the additional 'or P8' accepts reasonable risk.

  2. When the confidence of detection is very high, the risk is acceptable, thus the additional 'or C2' accepts reasonable risk.

The variable definition statement:

DETERMINE_QUALIFIER P8 MORE_STRINGENT_THAN P7

implies that:

  • when P8 evaluates to True, then P7 must also evaluate to True

  • When P7 evaluates to False, then P8 must also evaluate to False

example for confidence values and probability values
Define qualifiers as:
P8= prob <10^-8
P7= prob <10^-7
C2=99% confidence (1% error)
C1=90% confidence (10% error)

The statement

example of confidence and probability
SUITABLE Freeway EXCEPT WHEN Pedestrian@P8,C2

evaluates to True if and only if Freeway=True and (Pedestrian=False or P8=True or C2=True) implying that freeways are included unless there are no pedestrians with high probability or detection confidence is very hight.

The statement

example of confidence and probability
SUITABLE Freeway EXCEPT WHEN Pedestrian@P7,C1

is less stringent and thus accepts more situations.

The concept of qualifiers is well defined for both environment and engineering metrics. Nevertheless, engineering concepts should not be included in ODD specifications.

This approach implicitly assigns qualifiers in general, and P-values in particular, to constraints. For each constraints, all qualifiers are assigned a value, as part of the requirement to assign values to all variables. As such, semantically it is possible to group situations based on their associated qualifiers. For example, the following grouping is possible to identify rare situations of interest:

example selecting all constraints in ODD1 where there are pedestrians or the qualifier P8
SELECT * FROM constraints WHERE ODD1 AND (Pedestrian OR P8)

Further, the following grouping is possible to partition situation according to their importance:

extended example
SELECT * FROM constraints WHERE ODD1 GROUP BY Pedestrian,Pi FOR i in 1..8

In other words:

  • Qualifiers extend the baseline ODD to include additional situation for which the risk is acceptable.

  • Multiple levels of stringency can be used to represent a corresponding appetite for risk.

An approach to define semantics for “fuzzy” situations. The goal of uncertainty representation is to determine in or out of ODD:

  • Convert an otherwise “fuzzy” or “uncertain” situation into a black-and-white true/false determination.

  • Qualifiers are the specification of the conversion rules.

  • Data supporting evaluation of qualifiers originates from specific environment conditions (not from sensors).

  • Qualifiers are optional at the discretion of the ODD author.

  • Adding Qualifiers Increases Level of Detail (LOD).

For representing uncertainty, there are two major options:

  • Pure True/False Approach: Definitions should not contain any uncertainty qualifiers, e.g.:

    [green]#SUITABLE# Freeway [green]#EXCEPT WHEN# Pedestrian present OR IcyRoad
  • Qualified True/False: Increasing LOD using qualifiers, similar to refined road types, e.g.

    [green]#SUITABLE# Freeway [green]#EXCEPT WHEN# Pedestrian@P8 OR IcyRoad@P3.
Generic Qualifiers Approach
  • Qualified True/False: Adding qualifiers provided as input, similar to road types, e.g.

    [green]#SUITABLE# Freeway [green]#EXCEPT WHEN# Pedestrian@Q1,Q2,Q3 OR IcyRoad@Q4,Q5,Q6.
  • Qualifier Definition Statements, e.g.

    DEFINE(QualifierInterpretation, P1,”Probability of Occurrence <10-1”)
    DEFINE(QualifierInterpretation, P2,”Probability of Occurrence <10-2”)
    DEFINE(QualifierRelation, P2, MoreStringent, P1)
Multiple Qualifiers

Uncertainty Level Of Detail (ULOD)

Example: How do we represent fuzzy “expectation”, e.g. likelihood of drivers running red light:

  • Driving culture has a major impact, e.g. India vs Italy vs Germany

  • Location has major impact, e.g. busy city with lights in sequence vs rural roads with few lights

  • Timing has major impact, e.g. after transition green to red.

  • Traffic conditions have major impact, e.g. during traffic jam.

7.3.3. Multiple Qualifiers Q1,…​,Qn

Integrating Engineering Qualifiers

Approach supports Engineering Qualifiers, e.g. ML Confidence Values “Ci”:

  • Qualified True/False: Adding qualifiers provided as input, similar to road types, e.g.

    [green]#SUITABLE# Freeway [green]#EXCEPT WHEN# Pedestrian@P1,C2 OR IcyRoad@P2,C1.
  • Qualifier Definition Statements, e.g.

    DEFINE(QualifierInterpretation, C1,”Confidence >0.9”)
    DEFINE(QualifierInterpretation, C2,”Confidence >0.95”)
    DEFINE(QualifierRelation, C2, MoreStringent, C1)
    • SUITABLE Freeway EXCEPT WHEN Pedestrian@P8,C2 OR Cyclist@P5,C1 OR IcyRoad@P3,C2.

  • Language Requirements: Introduce a “DEFINE()” construct with a list of “relation types”. [_52]

8. Queries

While ODD definitions can be done by OpenODD files alone, queries and metrics on ODDs may sometimes relate only on information given in the OpenODD file alone, but sometimes also rely on external data sources (e.g. databases providing details on road types and geometries or weather conditions in a certain target country). While the technical access to external data sources like files and databases is performed via standard libraries, there must be a means to formulate queries like "select all roads with at least two lanes of at least 2.0 m lane width". This requires a query mechanism and an appropriate query language syntax (e.g. similar to the well-known SQL language). But also the use case of querying details on an ODD itself needs to be supported (e.g. for reporting or metrics), which can be achieved by the same query language.

8.1. User Stories for queries

8.1.1. Scenario Editor

Inclusion of a scenario within the ODD

As a scenario editor, given an ODD, I need to determine if a scenario lies entirely within the ODD.

Example

Approach

Given an ODD specification and a scenario (e.g. in OpenScenario language), check whether or not (Boolean answer) the scenario is entirely within the ODD boundaries.

Form the intersection of the scenario and the ODD (note that one has the appearance of a line whereas the other has the appearance of an area or a space, so additional considerations like giving a width to a scenario by making a shift from concrete to logical scenarios, or partitioning the ontology space are necessary in preparation). If the intersection is equal to the whole scenario, then the scenario is entirely contained within the ODD.

Note: There are also related metrics in the context of these user stories, see section on metrics.

Partial inclusion of a scenario within the ODD

As a scenario editor, given an ODD, I need to determine if a scenario lies partly within the ODD.

Example

Approach

Given an ODD specification and a scenario (e.g. in OpenScenario language), check whether or not (Boolean answer) the scenario is entirely within the ODD boundaries.

Form the intersection of the scenario and the ODD (note that one has the appearance of a line whereas the other has the appearance of an area or a space, so additional considerations like giving a width to a scenario by making a shift from concrete to logical scenarios, or partitioning the ontology space are necessary in preparation). If the intersection is not empty, then the scenario is at least partially contained within the ODD.

Which part of the scenario lies inside the ODD?

Similar to the example above, the part of the scenario that is not in the intersection is the part outside the ODD.

Which particular attribute or condition of a (piece of a) scenario caused it to be outside of the ODD?

Performing the intersection operation between ODD and scenario attribute-wise reveals which attribute(s) cause the ODD violation: It is the attribute(s) for which the intersection does not contain the entire scenario.

As a scenario editor, given an ODD, I need to determine if

(a) a scenario induces a violation of ODD attribute boundaries,

(b) which attribute boundaries of the ODD may be violated by the scenario,

(c) by how much or

(d) how long they may be violated#, which are important information when judging the fall-back or Minimal Risk Manoeuvre capability of an ADS.

Note: Cases (a) and (b) are queries (providing Boolean answer or selection or attributes or values for an attribute) and therefore discussed in the section on queries. Cases (c) and (d) are metrics (providing a real number or similar as an answer) and therefore discussed in the section on metrics.

Example

Approach

(a) Does a scenario include an ODD violation (i.e. at least a part of it lies outside the ODD)?

(a) Form the intersection of the scenario and the ODD (note that one has the appearance of a line whereas the other has the appearance of an area or a space, so additional considerations like giving a width to a scenario by making a shift from concrete to logical scenarios, or partitioning the ontology space are necessary in preparation). If the intersection is smaller than the whole scenario, then the scenario is not entirely contained within the ODD, so there must be at least one part of the scenario where at least one attribute violates the boundaries given by the ODD.

(b) Which attribute violates the ODD limit, and over which time interval of the scenario?

(b) FPerforming the intersection operation between ODD and scenario attribute-wise reveals which attribute(s) cause the ODD violation: It is the attribute(s) for which the intersection does not contain the entire scenario. The part of the scenario which is not in the intersection related to that attribute is the interval for which the boundaries are violated.

8.1.2. Test engineer

Inclusion of a scenario within the ODD

As a test engineer, I would like to know that a scenario that I have created is within the ODD.

See identical user story for role Scenario Editor above.

Inclusion of given aspects from ontology / domain model within the ODD

As a test engineer, I would like to understand which element of my driving ontology/domain model is in or out of my ODD

Example

Approach

Given an ODD and the ontology on which the ODD is defined, which attribute ranges / choices of the ontology are inside or outside the ODD.

Form the intersection of the whole ontology and the ODD. The part of the ontology that is in the intersection is part of the ODD. The part of the ontology that is not in the intersection is the part outside the ODD.

Note: This query depends on the completeness of the ODD specification. If the ODD specification does not say anything about a certain attribute that is defined in the ontology (e.g. weather) then it depends on the overall assumption permissive vs. restrictive how this shall be interpreted. In case of permissive, the whole range given for this attribute by the ontology is inside the ODD; in case of restrictive, nothing of it is inside the ODD - which can make the ODD unusable in practice (e.g. vehicle cannot operate in automated mode under any weather conditions).

8.1.3. Data scientist

Coverage by the ODD or some ODD aspects by a scenario library

As a data scientist, given a scenario library and optionally a distribution of parameter values to concretize these scenarios, [green]#I need to determine

(a) which aspects of the ODD are covered, so that I can

(b) assess which scenarios need to be added to support effective ODD testing.#

The question about aspects is not unambiguous. It can address the questions, which attributes (that are known from the underlying ontology) are explicitly addressed by the ODD specification (see metrics on ODD completeness in Section 6), but also which values for each attributes that are covered by the scenario set, or even which value combinations are covered. And if attributes are not covered by a set of scenarios, this may have different reasons: For instance, if a certain type of simulation works with simplified geometric descriptions of the road network and does not simulate any light or weather conditions, we cannot expect it to address attributes like weather or light conditions from ODD specification, and there is basically no way to fix this. Other simulations may be able to simulate these attributes, but they have not been considered explicitly but clamped to fixed default values. Even the opposite case can happen, that a simulation addresses values that are not explicitly specified by the ODD specification: for instance, almost any sort of simulation will have to make an assumption on the lane width - but there are ODD specifications that do not mention the lane width at all.

Even more complicated is the second question, which scenarios would need to be added to achieve completeness. It may be possible to point out combinations of attributes that still need to be covered, based on the given ODD, the existing scenario library and some completeness criteria. But even then it is not straightforward to (automatically) construct the most efficient selection of missing scenario so that the completeness goal will be achieved.

All of this is beyond the scope of the OpenODD language and therefore not elaborated here any further. The use cases given above for the role of the Scenario Editor may give some hints how to implement such queries (they be extended from a single scenario to a scenario library by forming the union set of all scenarios in the library.)

Note: In the same context, there are also related coverage metrics described in the section on metrics.

Inclusion of a scenario within the ODD

As a data scientist, given an ODD, I need to determine whether a specific scenario is included in the ODD, so that I can determine whether it needs to be included in the assessment of results.

See identical user story for role Scenario Editor above.

8.1.4. Inclusion of a specific event or condition occurrence within the ODD

As a data scientist, given an ODD, I need to determine whether an occurrence of something during a scenario is allowed by the ODD, so that I can determine whether the resulting vehicle behavior needs to be considered in the evaluation of correct / incorrect behavior (if a situation is in the ODD then we would expect the vehicle to handle it correctly, but otherwise the behavior of the vehicle is unspecified).

Note: The purpose is not to describe or monitor the vehicle’s intended or actual behavior; this would be beyond the scope of OpenODD. It is merely to check whether a certain occurrence in the scenario (in particular, in the vehicle environment) belongs to the ODD or not. If it does not, a potentially resulting misbehavior of the vehicle under test would not count as a failed test, because outside the ODD the vehicle’s behavior is unspecified.

Note: Currently there are no plans to make events or actions part of the OpenODD language. So only thing that can be detected in OpenODD is a condition in terms of whether attributes of a scenario take certain values outside the ODD. More advanced queries may be possible using other means, like queries on OpenScenario scenarios, considering an OpenODD file as a comparison object.

Example

Approach

Let there be a scenario from which the current values of certain attributes (e.g. visibility, ego vehicle speed) can be determined for each point in time and on ODD that contains restrictions on the same attributes. We want to know for a given point in time whether the specific attribute was inside the boundaries given by the ODD or not.

Compare whether the current value of the attribute is inside or outside the range or choice defined by the ODD.

8.1.5. Statistical distribution of parameters mentioned in the ODD specification

As a data scientist, given a scenario library, I need to determine (a) the statistical distribution of scenario parameters that occur in the ODD definition.

Note: This is going beyond the main scope of OpenODD. Only some outlines for a specific example can be given here.

Example

Approach

Assume that the ontology has some attribute with a discrete value domain (e.g. weather with values {sunshine, rain, snow, fog}), of which the ODD allows a subset (e.g. only sunshine, rain and snow, but not fog). Now it is interesting for several use cases to put relative probability tags to the values depending on their occurrence probabilities in a given target country, e.g. 50% sunshine, 30% rain, 10% snow, 20% outside ODD).

Use extension mechanisms to access statistical weather data for the given country (statistics must be available and accessible on some database server or in a file, which can be accessed through OpenODD extension library functions; it must also present the values sorted by compatible equivalence classes as used in the ontology.) Get the percentages for the different types of weather and associate them with the values of the attribute.

8.1.6. Safety Engineer

Coverage of the ODD by a situation catalog for Hazard Analysis (Functional Safety, SOTIF)

As a safety (SOTIF) engineer, given an ODD and a Hazard Analysis (acc. ISO 26262 / ISO 21448), I need to

(a) decide whether the the situation set underlying the Hazard Analysis covers the given ODD or

(b) list situations or attributes that are inside the ODD but not yet considered in the HARA situation set, so that I can assess the quality and completeness of the Hazard Analysis.

Note: This use case heavily depends on how the Hazard Analysis (and the underlying situation catalog) is represented in a specific tool, because there is no established standard exchange format for them. If the Hazard Analysis is performed using a plain EXCEL sheet, a custom solution would have to be scripted to retrieve and parse the strings for the situations and compare string-wise to the ODD / ontology entries. If a dedicated tool is used, it may provide an API that can be accessed through OpenODD extension library, or it may be able to work directly with OpenODD format and perform the query there. In the following only an outline for the proceeding can be given, assuming that the situation catalog of the Hazard Analysis can be retrieved in a format as if it was an ODD specification, and further assuming that both are defined over the same ontology.

Example

Approach

(a) Given an ODD and a situation catalog underlying a Hazard Analysis (containing discrete value choices for some or all of the discrete attributes appearing on the ODD specification, e.g. road types, and equivalence classes for some or all of the continuous-valued attributes appearing in the ODD specification, e.g. visibility). Are there any situations that are inside the ODD but not covered by the Hazard Analysis?

Form a cut set between ODD and situation catalog. If the cut set is equal to the ODD then there are no forgotten situations.

(b) Same as (a), but the question is: Which attributes and which attribute values for existing attributes are missing in the situation catalog?

Compare the list of attributes. If there are more attributes in the ODD than in the situation catalog, these are missing. Then compare for each attribute the value sets (or form attribute-wise cut set). The values from ODD that do not appear in the cut set are missing.

8.1.7. Infrastructure Operator

Vehicle warning via V2X when leaving the ODD (at vehicle operation time)

As an infrastructure operator, I want to have the ability to warn vehicles in advance if they are about to chose a route through my road network which most likely is exceeding their ODD

Note: This involves runtime decision making and V2X communication, which is outside the scope of OpenODD. Only an outline can be given how to detect whether a certain route is entirely within the ODD for a selected automated vehicle. It is assumed that the information about all relevant properties of a route (according to the attributes defined by the ontology, e.g. lane width or curvature or height of bridges can be retrieved from an external database via an extension library function and is in a compatible format that can be compared or intersected with the ODD (as shown above for the case of scenarios that are compared to an ODD); in particular, for each coordinate along the route all attributes according to the ontology can be retrieved.

Example

Approach

Given a route and an ODD specification of a certain vehicle: Does the route contain any portions that lie outside the ODD?

Form the cut set of the route (defined by the attribute value for each coordinate point along the route) and the ODD. If the cut set is smaller than the entire route there must be ranges of the route that are not accessible for this vehicle based on its ODD.

Navigation recommendations via V2X to choose route that does not violate the ODD (at vehicle operation time)

As an infrastructure operator, I want to give recommendations to automated vehicles which routes/roads/lanes to use depending on their ODD capabilities

Note: Giving routing recommendations is an advanced use cases going beyond the original scope of the OpenODD language and, thus, not outlined here. But the ideas from the use case above may serve as an inspiration how this task can be solved, provided that general knowledge on navigation and route-finding is available.

8.1.8. Development engineer

Runtime decision whether the vehicle is inside the ODD

As a development engineer, I need clear criteria how to decide at runtime whether I’m inside or outside of my ODD and to detect or predict that I am about to (unintentionally) leave the ODD so that I can implement detection and reaction functions for this case.

Note: The implementation of runtime decision algorithms is beyond the scope of OpenODD, but a typical use case based on an ODD specification in OpenODD language. In particular, runtime algorithms are bound to the restricted and sometimes inaccurate view of the environment through onboard sensors, self localization, HD maps and V2X, and the algorithms are limited by realtime constraints and onboard computational resources. Possible approaches include the detection or estimation of relevant attribute values of the ODD (e.g. lane width, weather) and/or looking up such properties from an enriched HD map that is available onboard, based on current location and prediction for the short-term future.

If it is for development and validation purposes sufficient to practice simulation of such scenarios on a host or a-posterior off-board evaluation of test rides with the actual vehicle, the same techniques as described above for the role Scenario Editor can be applied.

Testing against the ODD specification

As a development engineer I want to check tests, scenarios, recordings, etc. against the ODD specification.

Example

Approach

Is a scenario or recording from a test ride inside the ODD?

See similar use case given above for the role Scenario Editor.

What percentage is inside the ODD? (Note: This is rather a metric, not a query - see explanations in the subsection on metrics user stories)

Form the intersection of the scenario and the ODD. Divide the size of the cut set by the norm of the entire scenario.

Which parts of a scenario are outside the ODD?

See similar use case given above for the role Scenario Editor.

Which specific aspects cause it to be outside the ODD?

See similar use case given above for the role Scenario Editor.

9. Partitioning and equivalence classes

For many use cases it is mandatory to partition the ODD (as a whole or aspect wise) into equivalence classes. For instance, if lane width is a parameter of a road there could be three equivalence classes {narrow = below 2.0 m, regular = 2.0 …​ 3.0 m, wide = more than 3.0 m}. Equivalence classes are not necessarily formed on one single attribute, but sometimes also on combinations of several attributes (e.g. interstate highway means that road type is highway and lane width is at least 2.5 m) or conditional on other attributes (e.g. narrow is normally a lane with that is less than 2.0 m wide, but on a highway we say narrow even if the lane is less than 2.3 m wide). Such equivalence classes are a necessary precondition for several use cases around ODDs, e.g. some ODD metrics, in particular those who deal with the important aspect of coverage of the ODD by a set of scenarios. Sometimes, but not always, the partitioning is already made on the level of the ontology (e.g. if some ontology defines the strength of rain not in a unit like mm / h, but directly in equivalence classes like light rain, moderate rain, heavy rain). But this is not always the case and not always are the existing classes suitable for the purpose of some metrics, so this ODD extension provides means to create complex partitioning definitions on top.

Jason review comment: to me, the extension aspect of queries might only apply to external databases. query of the ODD itself is more of the toolchain or implementation tasks, rather than an extension of the language concept. The query properties of the ODD itself is already embedded within the WP2 language concept.

10. Exposure and probability

Exposure refers to the uncertainty of an occurrence. Probability is the method of quantifying that uncertainty. "Probability of Exposure" refers to quantifying the uncertainty of the occurrence related to the exposure.

Not everything that can occur within an ODD is equally probable. For instance, foggy weather may be part of the ODD of some automated function, but is expected to be rare in comparison to sunshine, cloudy or rainy weather. It is not necessarily a part of the ODD specification to specify the exposure to certain situations or the probability of certain events to occur, but doing so can support many activities around safety and risk assessment of automated driving systems. For instance, the Hazard Identification acc. ISO 21448 requires to judge whether some risk is initially acceptable, which depends on the exposure to a situation in which the vehicle reaction can lead to an accident. The same considerations apply for the judgement of acceptability of the final risk, after all improvement and validation testing has been made. It is also a well-known strategy to assign different testing or simulation efforts to different scenarios depending on their probability to occur. Some considerations on this topic and the following one can be found in the section on Uncertainty.

11. Dynamic aspects

An ODD specification may tell that an ADS function is suitable on roads with a legal speed limits of 100 kph, but will not state how fast the ADS is allowed to drive at any particular instant (this belongs to the ADS function specification). But sometimes it may be necessary to refer to description of dynamic elements (typically other participants) in order to specify situations that are suitable or unsuitable for a certain ADS function. For instance, a city pilot may be unsuitable for streets with "frequent crossing of pedestrian". However we need to define what is "frequent crossing of pedestrians"? In case this is not included in the ontology, the ODD author may have to define this term. This entails that in some cases, language means to express the characteristics of road users' behavior.

For more details about the difference between ODD and scenarios please refer to Difference between ODD and Scenario.

12. Onboard decision making

An automated vehicle must be able to determine at runtime whether or not it is inside its ODD. If it is outside the ODD, safe operation is no longer guaranteed, because the AD function has not been designed and tested for this environment. Accordingly, the vehicle must then inhibit the activation of the automated driving function or, if the ODD is unintendedly and unexpectedly leaving the ODD while driving in automated mode, the automated function must be safely terminated. This can happen in many cases by alerting the human driver and handing over control, but in some cases (e.g. vehicle without a human driver, human driver fallen asleep etc.) there must be a "Minimal Risk Maneuver", e.g. to slow down the vehicle and pull over to the road shoulder.

To enable all of this it must be possible to determine (and even predict for the horizon of a couple of seconds or minutes) whether the car is inside the ODD and whether it is about to reach the ODD limits. This is challenging for many reasons:

  • The view of the car on its environment is limited by the performance of its onboard sensors and V2X capabilities and totally different from the "omniscient view" of an ODD specification author.

  • Algorithms must be fast (real-time requirement) and feasible with the limited on-board computing resources.

  • Some exceptional events (e.g. the detected presence of a pedestrian on a highway) may appear very suddenly and unexpectedly; furthermore, depending on the ODD definition, such events can be allowed inside the ODD as long as their statistical occurrence frequency remains under a certain bound (which is not trivial to decide from a car’s perspective, see topic Uncertainty).

To solve all of this is the AD function developers' task, and not in the scope of ASAM OpenODD.

However, the OpenODD language should provide extensions to support this job in the best possible way, for instance by supporting a formal notation of detection conditions that can serve to derive specifications for the runtime system or even code libraries that can be linked into the runtime software of the AD function, and also test cases to evaluate the performance of such functions in realistic scenarios. Also means how to annotate in enriched HD maps the fact that certain attribute values relevant to delimit the ODD are present or absent (e.g. certain lane width, certain traffic signages or V2X facilities etc.) can support runtime decision making.

This kind of support will have to be elaborated in the standardization phase and could be implemented by a specific add-on library or by general requirements to the ASAM OpenODD language.

13. Annex to ASAM OpenODD

13.1. Compatibility with OpenX Standards

13.1.1. Integration with Scenario-based Testing Workflow

Summary: OpenODD language and format can be viewed as a stand alone language for several use case. In addition, there are many use cases where the format needs to easily be merged and combined with other component , in order to create a complete set of a test case ( or test cases ). When one looks at these test cases, the simulation and/or virtual testing world should be referred to. The next sections will describe several use cases, and the from these the requirements on the format/language will be derived. The key requirement is that that while OpenODD format is defining the ODD boundaries, it has to be integrated with the full testing system that interacts with the environment, and compares it to the ODD. Several examples of use cases:

The same scenario ( expressed on OPENScenario 2.0) need to be tested with different ODDs. This the scenario code itself, and the test specification need the ability to have a generic reference to an ODD, with assumed taxonomy structure The scenario and test code need the ability to match current environment conditions to the on specified on the ODD. The Scenario need to respond to events of environment changes, or ODD conditions changes. The Scenario need to respond to HMI request and flags, and match them again current ODD definitions and current environment conditions. The Scenario need the ability to observe actors, and match them against current ODD definitions in place.

Example syntax:

ODD expressed using OPENScenario 2.0 Enumerated types and constraints: As the ODD taxonomy and definitions are actually expressed in OSC 2.0 language, any scenario code can be easily merged with these definitions. ODD values, struct members and other can eb easily referenced and compiled together.

Example:
The ODD is defined as a struct entity, and can be used anywhere. ODD boundaries constraints are applied to struct members.

import odd_defs.sdl     # the ODD type declarations/taxonomy
# define odd_1 to be a subtype of any_odd, with extra constraints
# restricting the permissible conditions

struct odd_1: any_odd:
    keep(road_type in [motorway])
odd_1 is not an entity that can be addressed in used in the scenario code.

13.2. Terminology of the uncertainty representation

"Uncertainty Types" relevant to ODD:

  1. Uncertainty of Occurrences (past or future),

  2. Uncertainty of Environment (fully or partially observed),

  3. Belief in occurrence or environment

Term Definition Example Link

Probability of occurrence

Given an "event generation process" and a binary classifier labeling an event as "1" for every occurrence, it is the count of events labelled as 1 divided by the total number of events, as the total number of events asymptotically reaches infinity or sufficiently large

example

Belief in occurrence

It is an estimate of the "probability of occurrence"; The "probability of occurrence" is the "ground truth", and the "belief in occurrence" is the estimate.

Qualitative belief in occurrence

Whereas the true probability is always a numeric value, we can choose to represent belief of occurrence in terms of ranges For example, we can say that when the probability is <1% it is "low", and >99% it is "high".

Extrinsic representation

represent uncertainty in methods which are independent of data, e.g. mathematical expressions, formulas, etc. For example, an equation, .e.g gaussian distribution, or other symbolic representation not accompanied with data.

Intrinsic representations

represent uncertainty using the data, typically statistical methods, e.g. confidence intervals, explained variances, effect size, etc.

data lineage

traceability: the conduit through which data and its interpretation has passed; it is a chain of owners (or custody) and methods applied in each step

source credibility

the credibility of the source, as defined by the person using it

data credibility

the reliability of information source and the lineage

numerical precision

the level of detail in specification of a number, e.g. number of decimal points

numerical error

the numeric difference between a estimation and reality

relative numerical error

it is a ratio: "numerical error" divided by the number representing reality

numerical accuracy

a statistical statement on the error, e.g., confidence interval

uncertainty in environment

given measures/metrics representing the environment, it is a statistical statement on the error, e.g., confidence interval for each of these measures/metrics

examples

classification accuracy

number of correct classifications divided by the total number of items classified i.e. number of (true positive + true negatives) divided by the total number of items

false positives rate"

well defined for binary classification only; the number of incorrectly classified "1" divided by the total number of "0" items per ground truth

false negatives rate

well defined for binary classification only; the number of incorrectly classified "0" divided by the total number of "1" items per ground truth

precision, positive predicted value

the number of correctly classified "1" divided by the total number of classified (=predicted) "1"

recall, true positives rate, sensitivity

well defined for binary classification only; the number of correctly classified "1" divided by the total number of "1" items per ground truth

specificity, true negatives rate

well defined for binary classification only; number of correctly classified "0" divided by the total number of "0" items per ground truth

negative predicted value

the number of correctly classified "0" divided by the total number of classified "0"

false discovery rate

the number of incorrectly classified "1" divided by the total number of classified "1"

false omission rate

the number of incorrectly classified "0" divided by the total number of classified "0"

belief in safety

the confidence in a statement "vehicle is safe to operate in an ODD" relies implicitly on defining an "unsafe occurrence" in the context of that ODD and estimating the "occurrence probability" of that unsafe event.

scenario weight / probability of scenario

When averaging results across multiple scenarios, we need to know the individual weights. Clearly, scenarios which occur more frequently should have more weight than rare scenarios. Assuming that the scenario is generated from a data driven approach, then

crash probability for scenario

  1. non-determinism: when the results are not-deterministic, then measuring the probability of occurrence requires performing multiple runs, each regarded as an separate "event", counting the "crash occurrences" (label=1 for 1mm vs 1m apart is important) and computing the ratio of occurrence counts to the total event counts

  2. debuggable software: when the results of a scenario are deterministic, then the probability of a "crash" occurrence is either zero or one; this is a very bad estimate if we use parameterized scenarios for an ODD e.g. the pegasus "logical scenarios", we can measure probabilities because we can perform multiple runs and count fraction of occurrences. simulation needs to reproduce the non-determinism of the real world

  3. when events are rare, we need too many runs to obtain a stable result; for example for events with probability of P5 \~ 1e-5 we need 100,000 runs with different combinations.

  4. the ODD should provide guidance for parameterize the scenarios, in other words

probability of logical scenario

This is the probability of an occurrence called "scenario", which requires identifying (=labeling) scenario from many driving hours across a large number of drivers.

  1. need to specify parameter ranges to get non-zero probability

  2. the baseline count of events (for probabilities) can be defined as the total driving segments; e.g., for scenarios with 10 second durations, a single driving hour represents 360 events, and a 20 second scenario is regarded as two events

  3. the "probability of scenario" is therefore the ratio of the number of 10-second driving segments for which the scenario is detected (=labeled "1") divided by the total number of 10-second driving segments

  4. scenarios probability can be estimated (=belief) using tools and products for classification of scenarios from naturalistic driving data, e.g. from databases such as SHRP2

13.3. Probability of Occurrence examples

13.3.1. The probability that a rolled dice will land on number "3"

  1. Given a die and a method for rolling it

  2. and given a method for labeling of "1" when the landed number is "3",

  3. it is the number of "1" labels divided by the total number of rolls

13.3.2. The probability that a driver will run a stop sign

  1. Given an intersection with a stop sign and a collection of vehicles passing through it

  2. and given a logic for labeling of "1" when the vehicle is running a stop sign

  3. it is the number of "1" labels divided by the total number of passing vehicles

Note that these examples assume ideal labeling methods.
  • Accounting for imperfect labeling requires adding a conditional probability.

  • A full example of doing so is at the end of this doc.

13.3.3. probability of pedestrians on a road section" for a time period:

the "probability of occurrence" for events satisfying "pedestrian on road":

  1. given a specific road section and a collection of vehicles passing through it for a given period of time,

  2. and given a logic for labeling of "1" when "pedestrian is on the road" when the vehicle is passing

  3. it is the number of "1" labels divided by the total number of passing vehicles

Rule for correct counting:

  1. a vehicle passing a specific section multiple times is labeled and counted for each pass separately.

  2. a vehicle passing multiple sections is labeled and counted separately for each section

  3. aggregation over the time periods needs to be for similar time of day and other traffic characteristics, e.g. not mix 3am with rush hour traffic

  4. weighted averaging should be done based on total count of events, i.e. vehicles passing, in each section, i.e. sections without passing vehicles should be excluded

  5. "conservative assumption": the assumption that "the 1 labels occur independently" reduces the overall probability estimate as compared to situation where the labels are correlated

13.3.4. probability of pedestrians on a road section" for a time period for a specific ODD

Counting should only include events consistent with the ODD.

Comments and interpretation:

  1. The "probability of pedestrians on freeway in Germany" is associated with a time frame implicitly requires aggregating across all freeways in Germany for that time frame.

  2. When we don’t have perfect data, then we must estimate a number between 0 and 1 representing that probability, and hence we have a "belief that a pedestrian will be on the freeway in Germany"; the above concepts of numerical error and accuracy apply to that estimate.

  3. Estimating probability of "pedestrians on A-9", for example, requires aggregating estimates across all its sections across multiple time periods.

    1. If we know, for example, that we have a single pedestrian accident on A-9 freeway per year,

    2. and we know the number of vehicles pass through the section of the accident annually, then we know the probability for that section is 1 divided by that number per year.

    3. For all other sections of the road we get probability of zero. We can then average the results using event counts as weights.

  4. For freeways which are not used the denominator (=count of events) is zero (no cars passing), and thus must be excluded from analysis.

  5. The probability definition, and its estimate, requires counting, and thus it requires well defined counting rules.

    1. Clearly, the count of both events and "1" labels over longer durations will be larger as compared to counts for shorter durations.

    2. However, the ratio of the two count should change only slightly when that probability or its estimate is stable.

  6. When in a traffic jam, drivers exiting their cars are pedestrians on the road, but HARA/HazOp should not label them as "1".

13.4. Uncertainty in environment examples

13.4.1. Example: one measure of "uncertainty with pedestrians in my environment":

given a subject vehicle and a pedestrian in its path, and given the measure of the distance of the pedestrian from the vehicle, it is the confidence interval of the distance estimation error

13.4.2. Temporal uncertainty

How temporal uncertainty should and can be represented in an ODD will be part of the standardization project.

13.4.3. Weather uncertainty

Given a proposition specifying a weather condition, such as "rain" or "icy road", we would like to specify the degree of uncertainty of the specific condition at each location.

Example: The forecast is for rain in downtown. There is uncertainty regarding the exact measure for each specific location. In one street we can have 10%-20% more precipitation than in other streets. Similarly, In one intersection we can have a 10%-20% larger (or smaller) average droplet sizes.

We need to have a definition of "rain" in terms of "average droplet size" and "average precipitation". We can define the proposition "rain" to be true if "average droplet size">min_value, and "average precipitation">min_value. We can also define the expected range of values for each.

The information about expected average must be provided from outside the vehicle, because the vehicle is only able to measure at specific points, and cannot determine the average.

13.4.4. multiple views of uncertainty

visually describe uncertainty by integrating multiple views or perspectives of the representation (extrinsic or intrinsic)

related terms:

  1. "side-by-side view": comparing the same measure in multiple views side-by-side, typically visually

  2. "coincident view", or "combined view": comparing the same measure in the same view, by transforming onto a combined view, typically visually in a single chart

An example of "coincident view", estimated distance of pedestrian from the vehicle:

            d2       d1
            d1       d2
  . -----10-------20------
            d1
            d2       d2
                    d1

13.5. Open Issues

13.5.1. Format

t.b.d

13.5.2. Metrics

t.b.d

13.5.3. Uncertainty

The approach of simple propositional semantics in which propositions with optional qualifiers if very powerful: . It defines simple building blocks which can be assembled into very complex ODD specifications. . It supports multiple levels of abstractions, whereby the most abstract level includes only True/False propositions, such as 'Freeway' (easy to define for regulators), and additional definition expression increase the level of details. . Qualifiers further increase level of detail, and enable adding acceptable risk to the ODD thus providing support for safety engineering. . Qualifiers which represent environment uncertainty must be evaluated outside the vehicle and provided as input to the driving logic via V2X communication options. In contrast, engineering qualifiers not representing environment uncertainty can be evaluated on-board the vehicle. . Exchanging ODDs across manufacturers is possible by limiting the specification to the appropriate level of detail used in INCLUDE files. For example, different manufacturers may have difference C-values (confidence) in their testing while still conformant to the ODD specification which exclude those C-values. . Emergent situations are handled uniformly because the specification is "stateless" and does not include time. However, the language enables enrichment for engineering purposes, e.g. by adding behavior specifications, without adversely impacting conformance or exchangeablity.