Disclaimer
This document is the copyrighted property of ASAM e.V. In alteration to the regular license terms, ASAM allows unrestricted distribution of this standard. §2 (1) of ASAM’s regular license terms is therefore substituted by the following clause: "The licensor grants everyone a basic, non-exclusive and unlimited license to use the standard ASAM OpenODD". |
1. Foreword
The Association for Standardization of Automation and Measuring Systems (ASAM) is a non-profit organization that promotes standardization for tool chains in automotive development and testing. Our members are international car manufacturers, suppliers, tool vendors, engineering service providers and research institutes. ASAM standards are developed in work groups, composed of experts from our member companies. Our standards enable easy exchange of data or tools within a tool chain. ASAM is the legal owner of these standards and responsible for their development, maintenance, distribution and marketing.
ASAM standards focus on defining data models, file formats, communication APIs, software component APIs, and communication protocols for the data exchange between research, development, and validation systems. ASAM standardization seeks to incorporate requirements from many different global viewpoints and produce an efficient interface.
2. Introduction
2.1. Overview
Safety is fundamental to the development of public trust and acceptance of Connected and Autonomous Vehicles (CAVs) and their on board Automated Driving Systems (ADSs) and to enable deployment of automated driving. Safety of CAVs has two aspects: safe design and safe use of the system. In order to ensure safe use of the system, it is important to convey the knowledge of the true capabilities and limitations of the ADSs to the users to prevent misuse of the systems.
In order to establish the true capabilities and limitations of an Automated Driving Systems (ADSs), we need to first define their Operational Design Domain. An ODD refers to the operating environment (road type, weather conditions, traffic conditions) in which a vehicle can drive safely. For example, for Low-Speed Automated Driving (LSAD) systems such as pods and shuttles, the ODD may include urban areas with predefined routes that include pedestrians and cyclists. On the other hand, for a motorway chauffeur system, an ODD may include a four-lane divided motorway and dry conditions only. The types of scenarios a vehicle may encounter will be a function of its defined ODD, making this fundamental to any safety evaluation and scenario identification.
- A more formal definition of ODD as defined by SAE J3016 (2018) states that "Operating conditions under which a given driving automation system or feature thereof is specifically designed to function, including, but not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics". -
In order to enable stakeholders to share, compare and re-use ODD definitions, there is a need for standards to provide guidance to the stakeholders on both the attributes to be used for ODD definition and a format for defining the ODD using those attributes. BSI PAS 1883 (UK) provides a taxonomy for ODD. Additionally, ISO 34503 uses the taxonomy to provide a high-level definition format for ODD. While these standardization activities address the important needs of the industry, a gap still exists in the industry for an ODD definition format for simulation.
ASAM OpenODD is a representation of the abstract ODD specification in a more well-defined syntax and semantics which enables machines to interpret and perform the required analysis. Additionally, the ASAM OpenODD specification shall be measurable and verifiable for the attributes it specifies.
2.1.1. Motivation
An Operation Design Domain definition (ODD) shall be valid throughout the whole vehicle’s lifetime. The definition of an ODD is part of the safety concept of a vehicle. Depending on the current development step different information needs to be derived from such an overall ODD.
ASAM OpenODD focuses on a machine-readable format. The ODD must be represented so it can easily be used within simulation and other machine processed environments. The content of ASAM OpenODD will be derived from any abstract ODD, providing the information in a usable manner. For the purpose of using this ODD description for simulations and post-processing the format must fulfill the following requirements:
-
Searchable
-
Exchangeable
-
Extensible
-
Machine readable
-
Measurable and verifiable
-
Human readable
In the scenario based testing workflow ASAM OpenODD will play a very important role supporting the test description, defining the boundaries of what to test and achieving a good test coverage of the operational design domain and its borders.
For more details on scenario based testing workflows, consult the Paper released by the ASAM Test Specification Study Group. |
Using the ASAM OpenODD specification will ensure that it is possible to statistically quantify:
-
exposure of certain attribute values of the ODD,
-
the performance of an CAV and its systems against the ODD,
-
the tightening or expansion of the ODD over time.
ASAM OpenODD will be compatible with the wider ASAM OpenX suite of standards (OpenDRIVE, OpenSCENARIO and OpenXOntology). Furthermore, ASAM OpenODD will be consistent with the ongoing ISO activities (e.g. AWI ISO 34503). Compatibility of ASAM OpenODD will be achieved during a harmonization phase before the beggining of the standardization project.
2.2. Normative and non-normative statements and deliverables
This specification uses a standard information structure. The following rules apply regarding normativity of sections:
-
Statements expressed as requirements, permissions, or prohibitions according to the use of modal verbs, as defined in Section Modal verbs, are normative.
-
UML diagrams from the ASAM OpenODD UML model are normative.
-
Rules for ASAM OpenODD data structures in "Rules" sections are normative.
-
XML examples and use case descriptions are non-normative.
2.3. Conventions
2.3.1. Units
Unless stated otherwise, all numeric values within this specification are in SI units, for example:
-
position/distance in [m]
-
angles in [rad]
-
time in [s]
-
speed in [m/s]
Geographic positions are stated in the unit defined by the spatial coordinate system, for example, in accordance with WGS 84 – EPSG 4326 [1].
Some data elements allow to explicitly state the unit of a given value. If the unit is not given or if there is no means to state the unit, SI units apply. The following units may be used in explicit assignments:
Category | Description | Identifier |
---|---|---|
distance |
meter |
m |
kilometer |
km |
|
feet |
ft |
|
land mile |
mile |
|
speed |
meters per second |
m/s |
miles per hour |
mph |
|
kilometers per hour |
km/h |
|
mass |
kilogram |
kg |
metric tons |
t |
|
slope |
percent |
% |
These optional units shall be used for purposes of signage and speed indication only. They shall not be used as general units, for example, to define road geometry, etc.
2.3.2. Modal verbs
To ensure compliance with the ASAM OpenODD standard, users need to be able to distinguish between mandatory requirements, recommendations, permissions, as well as possibilities and capabilities.
The following rules for using modal verbs apply:
Provision | Verbal form |
---|---|
Requirement |
shall |
Recommendation |
should |
Permission |
may |
Possibility and capability |
can |
Obligation and necessity |
must |
2.4. Understanding ODDs
2.4.1. What is a ODD Specification?
An ODD defines the operating conditions an ADS is designed to operate in. This includes all ranges of roads, environmental conditions, part of the dynamic elements (mainly the ego designated speed) and types of traffic participants within a macroscopic traffic during all allowed weather conditions and time-of-day restrictions. The ODD will specify the domain and its borders the ADS is designed to safely operate in or needs to be capable of handling. This opens up a multidimensional space that needs to be specified.
According to SAE J3016 an ODD is defined as follows:
Operating conditions under which a given driving automation system or feature thereof is specifically designed to function, including, but not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics.
2.4.2. Operational Domain (OD) vs. Operational Design Domain (ODD )
A specific type of terminology misuse in practice is failing to distinguish between the real world and the ODD.
-
"ODD" refers to the intended ADS capability to handle operating conditions, and not the real world.
-
"Operational Domain (OD)" refers to what the world actually is, which might (in most cases will) differ from the ODD.
Thus, when the real world (OD) is outside the intended operational design environment (ODD), the vehicle has exited the ODD and is no longer in an environment it was designed to operate within. The difference between OD and ODD highlights the limitations of the ADS. Additionally, an automated driving system feature can have only one ODD. However, a vehicle can have multiple automated driving system features. Thus it is incorrect to say, for example, that a Level 4 highway feature has "multiple ODDs of day and night with fair weather" when what should be said is that the feature has "an ODD that includes both day and night in fair weather."
In all practical cases, an ODD definition will not be exhaustive enough to cover all attributes or occurrences in an OD. Therefore, it is essential to be able to define an ODD which is not only objective but which defines clear boundaries to enable ADSs to perform fallback manoeuvres to ensure safe operation.
2.4.3. Difference between ODD and Scenario
It is important to highlight that while ODDs and scenarios are related, they are not the same. As mentioned earlier, an ODD essentially defines the operating environment for which a system is designed. It may also be seen from the perspective of the end-user (e.g. city council authority) as the operating environment in which a system should be able to operate safely. It is essential that there is an overlap between the two perspectives of the ODD, manufacturer (or the system designer) and the end-user for ensuring the safe deployment of CAVs.
A scenario defines the behavior of various actors and entities in an ODD. Below the difference between scenario and ODD is highlighted graphically. [fig-simple-odr-odd] depicts a part of an ODD with single lane non-divided road which has passenger cars. Simple scenario with passenger cars depicts a scenario which illustrates the behavior of the vehicles in the ODD of [fig-simple-odr-odd]. In summary, once the behavior of the actors is defined in a part of an ODD, it becomes a scenario. This has no influence on the fact that a scenario is a stand alone entity regardless of the ODD definition and therefore can be defined independently
In Example ODD for the domain the graph on the left side represents the ontology or taxonomy the ODD specification can chose attributes from.
Figure 6a highlights the ODD attributes as per BSI PAS 1883. If an ADS has the ODD defined as Figure 6b, then the ODD comprises of a "divided single lane" road with a "T-junction" (scenery attributes of ODD). Furthermore, from the dynamic elements attributes, ODD comprises of "passenger cars". The difference between the ODD definition and the scenario definition is that the scenario definition also includes the behavior description of the actors, i.e. passenger cars.
2.4.4. The Scope and Usage of the ASAM OpenODD Language
Being a language for defining or describing ODDs is the main purpose of the new ASAM OpenODD language, but obviously there are many more activities in the domain of automated driving system concept, design, analysis and validation that can profit from a formalized language for ODDs. The proposed ASAM OpenODD language has been designed with many of these use cases in mind, and in the present concept phase there is no reason to consider the list as final and exhaustive. Nevertheless, usability for many stakeholders, including persons who are not specialists for complex computer languages, is a key concern for ASAM OpenODD development and entails the need to keep the core language simple and not to overburden it with lots of complex language constructs just to cover each and every potential use case. The solution chosen consists of a core language that is sufficient for the primary purpose of ASAM OpenODD, which is to describe an ODD. Around this core language, many extensions are grouped as shown in the ODD Definition and surrounding use cases. Also this list does not have to be exhaustive, according to its requirements, the ASAM OpenODD language will be designed with extension mechanisms so that future extensions can still be added. As shown in the picture, even the core part of defining ODDs makes references to external work: the ontology, on which each ODD description is based, can and should be referenced from external sources (in particular OpenXOntology) and specified in their respective standardized language.
The use of all extensions (grouped around the core language in the figure) is optional, but prepared in the structure of the language (including the ability to create "bindings" of functions to 3rd party implemented functions in any language like Python or C). For the concept phase, the following list of extensions is proposed, which could technically be released as "standard libraries" such as the standard libraries of programming languages like C form a basic ecosystem, on which users of the core language can rely.
The following extensions are proposed:
TODO add that not all chapter’s are fully flushed out due to the missing working group |
2.4.5. Completeness of an ODD
Since the information of the ODD is involved in the design, testing, release and so on, the completeness of the provided information becomes relevant. However, completeness should not be understood in an absolute sense, which means that no extra information can occur if a specific procedure is conducted. This is because of the open context, which makes it impossible to specify each and every aspect beforehand. Therefore, completeness is understood in a relative sense that requires us to define a reference against which the completeness of an ODD model can be measured. For the provision of such a reference, internal references as well as external sources can be used as a structure.
The completeness of the ODD results from the two aspects. This includes 1) completeness of the underlying ontology, and 2) completeness of the ODD description. To give a quick overview of the relation between the two, an ODD description language is a language format that has the capability of assigning the relevant ODD attributes to be either inside/outside its intended ODD boundary. However, these ODD attributes originate from an external taxonomy or ontology. Only by combining the ontology and an ODD description language can an ODD be produced. The BSI PAS 1883 on the taxonomy for the ODD of ADSs comments that from the perspective of the attributes/taxonomy, the ODD attributes list can never be exhaustive. This indicates that there is no 'complete' sense in terms of the underlying ontology that is used, which also resonates with one of the core language features that will be introduced in a later chapter (the extensibility of the underlying ontology). On the other hand, from the ODD language perspective, every piece of an ODD description must be complete, i.e, all the attributes from the underlying ontology need to be assigned and a binary (non fuzzy) boundary can be introduced across all the ODD attributes. To sum up, the underlying ontology can never be exhaustive, but the ODD description format needs to ensure every ODD is complete against the underlying ontology, and the permissive and restrictive concepts help to ensure completeness.
2.5. Use cases for ASAM OpenODD
In this chapter the use cases for the ASAM OpenODD are defined. In the development process the users of ASAM OpenODD were defined beforehand and the uses cases defined according to the users.
2.5.1. Users for ASAM OpenODD
Who will use and create the ODD and therefore use the OpenODD
-
The infrastructure operator includes Regulatory, Procurement, Safety Operator
2.5.2. User Stories
|
|
OpenODD Data Scientist Use-Cases, extending the ontology (i.e. need to support ontology by reference)
|
|
|
|
|
|
3. Relations to other standards
Name of the Standard | Description |
---|---|
ASAM OpenSCENARIO |
Description and exchange format for scenario descriptions |
ASAM OpenXOntology |
defines an ontology (and set of attributes) for the ASAM OpenX Domain |
ASAM OpenLABEL |
defines a standardized format for multi-sensor data labelling and scenario tagging. The format is based on a JSON schema |
ASAM OpenDRIVE |
standardized exchange format for the description of road-networks and the static entities on and along the road. ASAM OpenDRIVE is based on a xsd schema, using xml as an format |
ASAM OpenCRG |
a binary format to define road elevation or friction profiles |
ASAM ODS |
allows to store information in a database context and exchange it between tools. Based on a standardized meta-model (the base model) application-domain specific data models (the application models) may be defined. This may lead to consistent data sets e.g. combining ODDs with scenario based test definitions and corresponding test results for evaluations and long term storage. |
3.1. Positioning of ASAM OpenODD within ASAM activities
ASAM OpenODD is part of the ASAM simulation standards that focus on simulation data for the automotive environment. Next to ASAM OpenODD, ASAM provides other standards for the simulation domain, like ASAM OpenSCENARIO and ASAM OpenDRIVE.
ASAM OpenODD aims to describe the intended operating conditions (domain) an automated driving system is designed for. Part of the domain consists of the characteristics of the road network which can be derived from ASAM OpenDRIVE. A very simple road network is shown in Simple roadnetwork defined in ASAM OpenDRIVE. In this road network three incoming roads lead into a t-junction. This t-junction is made up out of several connecting roads.
As the road network only covers parts of the static elements of the driving environment additional information source may be needed (dynamic elements). The automated driving system also needs a specification of what kind of traffic participants it needs to be capable of handling. These dynamic elements and their behavior can be represented using ASAM OpenSCENARIO. A simple scenario with 4 passenger cars is shown in Simple scenario defined in ASAM OpenSCENARIO.
When analyzing the above two examples a few attributes of the ODD can already be identified. The automated driving system shall be designed to operate in:
-
divided single lane roads, with right hand traffic
-
a t-junction without traffic lights
-
passenger cars
This is shown in Example ODD for the domain
In addition to ASAM OpenSCENARIO and ASAM OpenDRIVE, ASAM OpenODD is also closely linked with the ASAM OpenLABEL standard, especially on the scenario tagging aspect. Figure below illustrates one example use case of the scenario tagging within ASAM OpenLABEL, it can be seen that each scenario is associated with corresponding tags from a bucket of tags. Based on such information, scenarios can be searched easily within the database. The link between ASAM OpenODD and the ASAM OpenLABEL scenario tagging aspect is that ODD tags form a major part of this 'bucket of tags'. The ASAM OpenODD attributes are closely aligned with the ASAM OpenLABEL tagging model. By utilizing such common properties, organizing and searching of scenarios and ODDs, and their dependencies can be effectively achieved.
3.2. References to other standards
Name of the Standard | Description |
---|---|
ISO 34503 (in development) |
Provides a taxonomy for operational design domain for automated driving systems, along with a high level definition format for use by regulators and non-coders. |
BSI PAS 1883 |
Provides a taxonomy that can be used to ODDs for automated driving systems |
SAE AVSC00002202004 AVSC Best Practice for Describing an Operational Design Domain: Conceptual Framework and Lexicon |
Aims to establish a best practice for Operational Design Domain description, establishing commonly defined terms and recommending a framework in which they can be applied. |
SAE J3016 (2021) / ISO 22736 |
Defines the term ODD and its use for classifying levels of autonomy for automated driving systems |
ISO 21448 SOTIF |
Provides guidance on defining the safety of an ADS system or feature for its intended functionality. |
ISO TS 5083 (in development) |
Provides guidance on safety of an ADS system (design and verification stage). |
4. Language and Format for ASAM OpenODD
The language and format chapter forms the core language concepts of the OpenODD. Every language development was made against a pre-identified requirement which was identified during the initial phase of the project. The content in this section will provide an overview of the overall development process, followed by the identified requirements clusters, and furthermore examples were provided for each of the requirements. These examples contain two different syntax proposals and the query semantics (where applicable). Thirteen different requirement clusters in total have been identified, which define the scope of the OpenODD format:
-
readability
-
composability
-
conditional statement
-
extensibility of ontology
-
binary boundary
-
class/metric/data type/ units
-
expression/abstraction
-
extensibility by function
-
operators
-
permissive/restrictive
-
language extensions(probability
-
uncertainty)
-
integration with scenario-based testing workflow.
From the thirteen requirement cluster examples, both the semantic and syntactic aspects of the language are covered.
The chapter includes:
-
Overview of the language concept development process
-
Requirements collection and requirement clusters creation
-
Main language concepts
-
Main language concept (extensibility of ontology) and extended language concepts (metrics, probability&uncertainty and extensibility by function) centered around the extensibility nature of the language
-
Integration with scenario-based testing workflow
4.1. Example syntax representation
During the concept project several syntax approached evolved and were proposed. The project decided to move forward with two of these proposal:
-
Syntax 1 is aligned with OpenSCENARIO 2
-
Syntax 2 aligns with the query semantics approach which evolved during the OpenODD concept project phase
4.1.1. Introduction into syntax 1
Modelling assumptions and principles for representing and using ODDs using OSC 2.0 language:
OpenSCENARIO 2.0 is an object oriented, aspect oriented, declarative, domain specific programming language. As such, it offers the full feature set of programming languages, in terms of operators, types and variables. It includes methods, structures, units, constraints specification and more. The examples are using OpenSCENARIO 2.0 syntax and constructs, with some assumption on a potential way to include and refer to ODD specification, in the same language in which scenarios are written, and using the same types, units and entities.
ODD definition:
We assume the full ODD ontology is defined in a type called ODD.
Sets of values are defined as enumerated types, for example:
type rain_type: [none, light, moderate, heavy] type wind_type: [calm, moderate, storm] type road_type: [city_road, motorway] Sub-hierarchies are defined as structs, for example: struct weather_type: odd_struct: rain: rain_type wind: wind_type The full ODD is defined by hierarchical composition: struct ODD: odd_struct: weather: weather_type road: road_type and so on…
odd_struct
ODD types are declared using a subtype of OpenSCENARIO struct called odd_struct. odd_struct fields may represent:
-
a physical value or range like speed
-
an enumerated value representing some classification, like light_rain
-
a Boolean flag like has_guardrail
-
a numeric value or range like number_of_lanes
-
a sub odd_struct representing a hierarchy
The current ODD is captured by top.odd which is an instance of odd_struct
constraints
Constraints are added to odd_struct, representing the limitations imposed by the ODD specification. They should be set-oriented. Only rarely does constraining to a specific value makes sense in ODD context. The synthetic constraint all_default_values() restricts all unconstrained fields to their default values, , implying that any restrictions on them will lead to contradiction. This is used when defining a restrictive ODD.
OpenSCENARIO 2.0 supports importing of packages (source files). Once imported, the types defined in these packages are available. This is the expected method to import either the ODD ontology/taxonomy , or ODD definitions.
Binding method to external library
The binding of a function defined in an external library to a method in OSC2.0 is done as follows:
def <method-name>(<arguments>):<returned-type) is external.cpp("function-name", "shared-object-name")
Once defined the method can be called to invoke the associated library function.
4.1.2. Introduction to syntax 2
The semantics of an ODD specification are defined as a mapping of all possible variable value assignments (continuous, integers, binary or enumerated type) to a True/False determination. Intuitively, it is equivalent to an SQL query selecting of all possible situations whereby the "WHERE" condition is provided by the ODD specifications: Situation returned by such query are included in the ODD; all other situations are excluded.
Each statement must start at a new line, but it may span multiple lines. When a statement spans multiple lines, the 2nd and subsequent lines must be indented.
Each statement must start with one of the following keywords: INCLUDE
, SUITABLE
, UNSUITABLE
, DETERMINE
, ADDCOND
,MEASURE
. In addition, the TAG
keyword can be used towards the end of certain station as a packaging mechanism to bundle a set of ODD attributes and form under a new ODD name. The keywords defined are not yet completed and need to be extended for cover all required functionalities.
Each statement is associated with specific syntax rules:
-
INCLUDE
- Include statements start with the keyword "INCLUDE" followed by a file name. Included files may be an ontology file or other ODD specifications, and there is no restriction with regard to the depth (or breadth) of the include tree. -
SUITABLE
/UNSUITABLE
- these two keywords are used at the start of the corresponding statements to indicate the permissive/restrictive nature applied to each individual statement. They have the following patterns (parenthesis denote optional keywords):suitable conditionSUITABLE <concept> (EXCEPT) WHEN <condition>
unsuitable conditionUNSUITABLE <concept> (EXCEPT) WHEN <condition>
-
ADDCOND
- it is used to add custom extension to an existing domain model or ontology, for example if user wants to add 'my_special_T-junction' to an existing ODD attribute 'T-junction',ADDCOND
can be used. The standard format for usingADDCOND
is:ADDCOND
<new_condition>
TO
<concept>
(AS
<relationship>
) whereby:-
<concept>
is defined by a previous statement, either in the same file or in an included file. -
<relationship>
defaults to "child-of" if the expression is omitted; otherwise, a specific relationship is added to the concept graph, this part is optional. Example: user would like to add 'my_special_T-junction' under the T-junction class, the format would be:
-
ADDCOND my_special_T-junction TO T-junction
-
DETERMINE
- this keyword is used to construct new classes/or to assign existing classes using a set of values. For example, is user wants to specify that 'my_special_rain' is when both the rainfall rate greater than 5mm/hr and the droplet size greater than 1mm diameter, the keywordDETERMINE
can be used to perform such assignment (see in the example below). The format follows the structure:DETERMINE
<concept>
WHEN
<expression>
whereby the<expression>
is an and-or-tree converting continuous variables/metric values to True/False values.
For example:define a constraint with a given rangeDETERMINE rain_heavy WHEN (rain_rate >5mm/hr) AND (droplet_size >1mm)
-
MEASURE
- this is used to assign measuring unit and data type to an ODD attributes. For example, to assign mm/hr as a measuring unit for rainfall rate, while the datatype is float, such operation can be achieved by using this keyword. The "MEASURE" statement follows the format as below :-
MEASURE <symbol> TYPE numeric | enum
If TYPE is "numeric" we have the following additional component to this statement:MEASURE <symbol> TYPE numeric UNITS <unit_symbol> [RANGE <min>,<max>] [DEFAULT <value>]
-
For the example listed above, the MEASURE statement would be
+
MEASURE rainfall TYPE float UNITS mm/hr
-
TAG
-TAG
is used to package a selected numbers of parameters and provide them with a new ODD name. For example if user need to combine ODD1 and ODD2 to form a new ODD3, this keyword can be used. It follows the format below:<conditions> TAG <new_ODD_name>
, where the conditions is the inclusion or exclusion. An example is shown below:SUITABLE ODD1 AND ODD2 TAG ODD3
4.2. Overview
This chapter focuses on the format of the OpenODD language. As illustrated earlier, the OpenODD language needs to satisfy various requirements from different target users. Such requirements will form the foundation of the language development, and will create a coherent storyline throughout this chapter.
To understand how this chapter is structured, it is important to first understand the overall development process the work package took during the project. In the early stage of the development, this list of requirements went through an expansion followed by a contraction phase. The expansion part built on top the identified use cases, and called for a further gathering of additional relevant requirements within the working group, this resulted in an extended requirement list. The group then evaluated the relevance of each submitted requirements, and provided necessary examples. The contraction part performed a clustering exercise on this extended list, many similar requirements were grouped together to form a 'requirement cluster', whose definition is the union of all its sub-requirements. Upon obtaining a list of requirement clusters, language concepts centred around each cluster were developed and proposed. Due to the nature of a concept project, the group decided to accept all the relevant proposals and suggestions (providing that the same concept proposal covers all the clusters). It will then be the responsibility of the standardization phase to consider the pros and cons of each concept and arrive at an unified concept. In addition to the language concept proposal, the group also utilized a more semantic-centred approach to illustrate each cluster, this is based on a neutral concept of query semantics, which is represented in syntax 2.
With the overall workflow introduced, readers should find the chapter structure easy to follow.
Section 4.2.1 provides the background and reference to the extended requirement list, the list also indicates group members' comments and those requirement entries that the group decided to be irrelevant.
Section 4.2.1 then introduces an overview of the requirement clusters together with their definitions which resulted from the extended requirement list.
Section 4.3 onwards provides detailed content associated with each of the clusters.
Such content includes:
-
a detailed definition of the cluster,
-
example challenge(s) the group proposed that can represent the cluster,
-
how the challenges can be tackled using the OSC2.0 language and query semantic,
-
how such challenge(s) can be tackled with different proposed and developed syntax.
By reading this chapter, readers will gain understanding of the scope of the language concept, the details of each requirements, and the approaches/proposals.
4.3. Requirements for ASAM OpenODD
4.3.1. Collected requirements list
During the runtime of the project the project group collected requirements, the requirements are clusters into categories. In total the categories are as follows
syntax |
rules of writing, Format |
semantics |
how to interpret the written |
process |
how to integrate and/or use the written |
The link above leads to the extended list of requirements which form the foundation of future OpenODD language & format development work. Please note that within this list, project members have voted that requirements 60, 56a, 56b, 54, 62, 61, 37 will not be considered for the first draft of the OpenODD Format.
4.3.2. Requirement Clusters Overview
The Requirements are being clustered in the following clusters
Nr. |
Cluster |
Description |
Human readability |
ODD language needs to be intended to be used by various stakeholders, and it will be used as input to testing and analysis as well as specification, communications, analysis and debugging. |
|
Machine readability and query |
The language needs to be machine readable, can be parsed using any text parsers, should enable queries for attributes of the ODD. UTF8 Test using DSL,XML,YAML,others. |
|
Composability |
The language shall support combining/re-using multiple ODD definitions into a new, wider ODD, and it shall provide the ability to create a named subset of ODD attributes |
|
Parameterization and Templating |
In order to enable flexibility in definition, and flexibility in usage for testing and definitions, as well as the ability to supply templates for ODD definitions for various usages , the language should enable a parametrization mechanism, where names are replacing values, and are resolve toward actual usage time. |
|
Conditional statement |
Language specification needs to support conditional statement or reduced ODD. This can be tackled by imposing constraints on the full operational range of specific attributes. |
|
Extensibility of Ontology |
The language shall support the addition of new attributes and values that are not in the original ontology. It shall support the establishment of standard libraries, e.g. PAS 1883. |
|
Binary boundary |
The language shall map every situation into binary in/out of the ODD, convert 'uncertain' or 'fuzzy' situation into binary. |
|
Class metric / datatype / units |
|
|
Probability / Uncertainty / risk |
The language shall support qualifiers to represent uncertainty,probability and risk as optional element |
|
Class Hierarchy / expressiveness / abstraction |
The language format needs to be able to express the class hierarchy provided by the attribute group (WP01), it needs to support inheritance, be consistent with the class categorises (keywords), data type and units. It shall support abstract ODD definitions by choosing different hierarchical levels. |
|
Extensibility by function |
The language shall be extensible by internal or external functions. |
|
Operators |
The language shall use a set of mathematical and logical operators, such as +, -, and, or, >, <, etc |
|
Permissive / Restrictive |
The language should take a mixed permissive and restrictive approach to its statements of what is inside the ODD. |
|
Integration in scenario based testing workflow |
The language shall supply the ability to refer to ODD definition from scenario definition, interpretable to create a full test description , and can be used for different purposes (such as requirements, checking, success criteria or testing) |
|
Compatible with OpenX Standards |
The language shall be compatible with other OpenX standards and ASAM ODS, share a common domain model. |
4.4. Concepts for the language and format of ASAM OpenODD
Of the 15 requirement clusters, some of them can be bundled together due to similar nature. Instead of mechanically introduce them one by one, the following sections will be divided into three main parts: section 4.3 contains the first group of clusters where little similarity can be drawn between them, in addition, the concept of mandatory and optional attributes will also be covered in section 4.3; section 4.4 contains those clusters with extensibility property, this includes those belong to the core language concepts (i.e, extension by function, extensibility of ontology), and also those outside the core language concepts (i.e, metrics and probability); section 4.5 will introduce the integration with the scenario-based testing workflow.
4.4.1. Mandatory vs optional Attributes
The role of mandatory and optional attributes can be illustrated in combination of another key language feature (permissive/restrictive, which will be introduced in Permissive and restrictive definition). To briefly introduce, permissive means everything that is not explicitly describe in the ODD will be permitted, whereas restrictive corresponds to hte opposite. Now given the following ontology
If a blank ODD is defined in permissive mode, all the above attributes are allowed, if the blank ODD is defined in the mode restrictive, none of the above attributes would be allowed. This means that the only required attributes of an ODD defined using ASAM OpenODD are the global permissive/restrictive mode, and the reference to the underlying ontology. However, although not mandatory in nature, in some cases the attributes which can be associated with numerical values are also recommended to be specified. In the above example, the lane dimension is one such example. Let us imagine an ODD with such an ontology and a global permissive mode, this means the system can handle any lane dimensions that are listed in the ontology. While this could be true it is unlikely an ADS with a conventional vehicle size can handle extreme narrow lanes such as 1.50m.
4.4.2. Human and machine readability
ODD specifications are intended to be shared across wide range of stakeholders such as regulators and test engineers, it would be used as input to testing and analysis, but at the same time to be used by humans for purposes of specification, communications, analysis and debugging of systems under development. This requires ODD specification to be understandable to both technical and non-technical users. In the meantime, ODD specifications should enable machine readability for purposes such as runtime ODD coverage checking, comparisons between multiple ODDs. The language is intended to serve as input to computer programs, that should be able to parse, analyze , query and interpret the ODD information. In particular, query has been identified as an important requirement within the project group. Query could be performed between a traffic situation (corresponding to the concept of OD mentioned in the introduction) and the corresponding ODD specification utilizing an ontological approach.
Example use case:
An example ODD states that motorway is only suitable when there is no rain, up slope is not suitable as the vertical geometry.
Example representations:
-
Syntax 1:
keep(road_type == motorway => odd_5.weather.rain == none) keep(geometry.vertical != up_slope)
-
Syntax 2:
SUITABLE Motorway EXCEPT WHEN Rain UNSUITABLE Up_slope
4.4.3. Composability
The ASAM OpenODD language format should be composable, (i.e., combine ODD definitions into a new, wider one or use inheritance mechanism). In addition, the ability to re-use a defined section of an ODD should also be possible, by providing the ability to create a named subset of ODD attributes that can be defined together. Unions and intersections are examples of composing two ODDs, in addition the negation of ODDs can also be composed with other ODDs. When selecting subclasses from a parent class, the dot operator can be used such that if b is a sub-class of a, then a.b will point towards the b subclass, such convention can be seen in both syntax examples below. From a metric perspective, the language shall allow building new metrics out of existing metrics, functions and operators. This will be covered in the extensibility section.
The ability of being composable will also enable comparison of ODDs and identfiying contradicting specifications in the ODD. This can be very helpful when comparing a vehicle ODD to a City ODD, checking wether the car is able to operate safely in the streets of the city.
In the following abstract example the underlying meaning of of composing ODDs is illustrated. As can be seen that the first tree of attributes (first ODD) differs from the second tree, by composing these two together the third tree (ODD) is obtained, which is the union of the two. Please pay attention to the conditions that the speed for highway and for urban roads are separate (these are inherited from the individual ODDs and are not mixed during the composing process).
In the following abstract example the concept of comparing ODDs is illustrated. The mentioned "City ODD" can be seen as requirements towards the capability of the ADS. ADS ODD will be the designed capability of the vehicle. By comparing the two ODD specifications the suitability of the ADS towards the City requirements can be established.
Result:
The City ODD spedifies that any vehicle there (if going in automated mode) shall expect and be capable of handling
other vehicles moving with up to 60kph
while the ODD specification of the automated vehicle states under development states that its own maximum speed
is 50kph
. This alone is not a problem for operating the vehicle in the city (ego vehicle speed capability and traffic speed capability are two different things); it is sufficient that the vehicle can cope with traffic up to 60 kph, even if it cannot reach that speed itself.
Be aware that case would be different if there were constraints on the achievable speed of our vehicle, for instance related to a highway settings, where in some countries traffic regulations require that only motor vehicles with a nominal max speed of at least 60kph are allowed to be present; then, a vehicle with a technical max speed of only 50kph would be explicitly forbidden to enter that highway.
But there are more differences between our vehicle’s ODD and the city ODD:
-
The vehicle can only handle up to
medium traffic density
. but the city’s ODD specifies that there can behigh traffic density
. -
The vehicle’s ODD specification states that is not capable of handling
cyclists
, where the city only states thatlorries
are not to be expected - so cyclists can occur. -
In terms of weather, the gap is that the vehicle can not operate in
fog
andsnow
which is according to the city ODD is to be expected.
As a result it can be concluded that the specified vehicle is not fully capable of operating in the given ODD of the city, and thus is not allowed to enter the city on automated driving (or, respectively, allow the human user to switch on the automated driving function while running in that city).
Note: In the section on queries there are more usage examples of such comparisons between a vehicle’s ODD and other ODDs, roadmaps or scenario sets.
Example use case: Compare and Combine
Given an ODD_1 which specifies that motorway is not suitable without restriction on the remaining classes on the ODD attribute hierarchy. Given another three ODDs (ODD_2, ODD_3, ODD_4) which specify wind conditions, rainfall condition and snowfall condition. The goal is to:
-
construct a weather ODD (ODD_5) by combining ODD_2, ODD_3 and ODD_4
-
construct a combined ODD (ODD_6), which takes the motorway ODD and weather ODD.
-
in addition, illustrate how to obtain the intersection, union and difference between two ODDs
-
Example syntax:
-
Syntax 1:
# ODDs mentioned in challenge: odd1: ODD: keep(road != motorway) odd2: ODD: keep(weather.wind != storm) odd3: ODD: keep(weather.rain in [none, light]) odd4: ODD: keep(weather.snow == none) odd5: ODD: # feature selection keep(weather.wind == odd2.weather.wind and weather.rain == odd3.weather.rain and weather.snow == odd4.weather.snow) # Since odd2, 3, 4 are orthogonal ( no overlap ) we could also create a union: odd5_1: ODD: # union keep(weather == union(odd2.weather, odd3.weather, odd4.weather) odd6: ODD: keep(road == odd1.road) keep(weather == odd5.weather) # recursive equivalence
-
Syntax 2:
# For combining ODD1 and ODD2 into ODD3, various options depending on the combination operation: INCLUDE example_ontology # this contains rain_type, wind_type, road_type, geometry, weather_type ADDCOND [none, light, moderate, heavy] TO rain_type ADDCOND [calm, moderate, storm] TO wind_type ADDCOND [city_road, motorway] TO road_type # ODD ontology modeled as a hierarchical composition of odd_structs ADDCOND [rain_type, wind_type] TO weather_type SUITABLE motorway EXCEPT WHEN rain UNSUITABLE geometry WHEN up_slope # ODDs mentioned in challenge: UNSUITABLE road_type WHEN motorway TAG ODD1 UNSUITABLE weather_type WHEN storm TAG ODD2 SUITABLE rain_type WHEN none OR light TAG ODD3 UNSUITABLE weather_type WHEN snow TAG ODD4 SUITABLE * WHEN ODD2 AND ODD3 AND ODD4 TAG ODD5 SUITABLE * WHEN ODD5 AND ODD1 TAG ODD6 # In addition, below illustrate different composabilities # Intersection: SUITABLE * WHEN ODD1 AND ODD2 TAG ODD3 # Union: SUITABLE * WHEN ODD1 OR ODD2 TAG ODD3 # Difference ODD1-ODD2: SUITABLE * WHEN ODD1 AND NOT(ODD2) TAG ODD3
-
-
Example use case: selective combination
Import a sample ODD as ODD1 which constraints on both road type and weather. Import another ODD (ODD2), construct a new ODD (ODD3) that contains the weather part of ODD1 and the whole of ODD2.
Example syntax:
-
Syntax 1:
import odd_lib/ODD1.osc-lib # declares ODD1 import odd_lib/ODD2.osc-lib # declares ODD2 odd1: ODD1 # as imported odd2: ODD2 # as imported odd3: ODD keep(me.* == odd2.*) # all features equal to odd2 keep(weather == odd1.weather) #select the weather sub-hierarchy
-
Syntax 2:
# It is important to differentiate ODD and ontology, within the statement # attribute without namespaces can be treated as coming from the ontology. # We could use namespaces for include multiple ontologies. INCLUDE PAS_1883.owl INCLUDE sample.odd AS ODD1 INCLUDE ODD2 SUITABLE ODD1.weather AND ODD2 TAG ODD3
Combining ASAM OpenODD files should not allow inconsistencies in the combined ODD description. Checkers and validators should be implemented in the toolchain, to avoid any inconsistencies. |
4.4.4. Parameterization and Variable definition
In order to enable flexibility in definition, flexibility in usage for testing as well as the ability to supply templates for ODDs, the language should enable a parametrization mechanism, where actual values are replaced by variable names, and are parameterized during actual usage time.
Example use case: Variable usage
Example on how to use variable top_speed_limit
in ASAM OpenODD
Example syntax:
-
Syntax 1:
keep (speed < top_speed_limit) # where the value of top_speed_limit is undefined, and will be defined once this statement is actually being instantiated or used
-
Syntax 2:
DETERMINE safe_speed WHEN speed < top_speed_limit SUITABLE * WHEN safe_speed
Example use case: Variable definition
Example on how to use variable rain
of type enum
in ASAM OpenODD, the actual value of the enumerations can be defined during usage
Example syntax:
-
Syntax 1:
type rain_type: [none, light, moderate, heavy]
-
Syntax 2
# define the constraint for rain, a constraint will either be True or False ENUM rain VALUES {no: WHEN mNoRain, low: WHEN mLowRain, medium: WHEN mMediumRain, heavy: WHEN mHeavyRain};
4.4.5. Conditional statement
Language specification needs to support conditional statement or reduced ODD. This can be tackled by imposing constraints on the full operational range of specific attributes.
When defining an ODD for a vehicle or a town, it might be necessary to define constraints according to certain conditions. This might be due to environmental conditions that degrade the capability of the vehicle or geographic specific conditions that have an impact on defined constraints.
Example use case: basic constraint definition
Examples showing how basic constraints are handled.
Example representations:
-
Syntax 1
odd1: ODD: keep(weather.rain in [heavy] => speed in [0..60]kph) keep(weather.rain not in [none, light] => road != motorway)
-
Syntax 2
SUITABLE rain.heavy WHEN speed < 60 UNSUITABLE Motorway EXCEPT WHEN rain.none OR rain.light
Example use case: add constraint definition
Adding constraint to an existing constraint
Example representations:
-
Syntax 1
odd1: ODD: keep(speed in [0..80]kph) keep(weather.fog == heavy_fog => speed in [0..40]kph)
-
Syntax 2
adding an additional Condition in Syntax 2SUITABLE speed > 10 EXCEPT WHEN rain.heavy SUITABLE rain.heavy WHEN (speed < 60 AND speed > 10)
Example use case: Apply basic constraints
The velocity in which an ADS system is operating is defined to be between 0 and 80kph. However, in the case of the a heavy fog ( assuming heavy fog is well defined ) , the ADS system is constrained to operate in velocity range of 0 to 40kph.
Example representations:
-
Syntax 1:
struct any_odd: max_speed: speed weather: odd_weather keep(max_speed < 80 kph) # restricted unconditionally # in case of fog, restrict further: keep(weather == heavy_fog => (max_speed <= 40 kph))
-
Syntax 2:
DETERMINE speed_high WHEN speed (> 40 AND =< 80) DETERMINE speed_low WHEN speed (=< 40) # MEASURE for specifying unit can also be combined with DETERMINE MEASURE speed UNITS kph UNSUITABLE speed_high WHEN heavy_fog SUITABLE speed_safe AND speed_low EXCEPT WHEN heavy_fog
Example use:
Given an ODD hierarchical tree, we would like to express that Motorway
is only suitable when there is no Rain
(conditional ODD statement),
within the Geometry
, Up-Slope
is not suitable, all Weather
conditions are suitable.
Example representations:
-
Syntax 1
odd1: ODD: keep(weather.rain != none => road != motorway)# no motorway in rain keep(geometry.vertical != up_slope)
-
Syntax 2:
SUITABLE Motorway EXCEPT WHEN Rain SUITABLE Vertical_geometry EXCEPT WHEN Up_slope
4.4.6. Binary boundaries
Any ODD definition requires the ability to have binary decision on any constraint, specified in the ODD.
The language will define the boundaries of the ODD specification. The language shall allow the mapping of every ODD terminal element into either inside or outside of the ODD in a binary manner by any external processor. This results in an ODD representation which can be evaluated to a set of TRUE/FALSE ODD terminal statements.
From a mathematical point of view such a representation relates to the ordered binary decision diagrams (OBDDs). From a programming point of view the OBDDs can be handled through boolean logical programming that allows to efficiently apply to OBDDs many logical operations (like conjunction, disjunction and negation) promoting composability of the language
Imposing restrictions to the possible combinations of ODD elements (concurrent evaluation of specific ODD elements to TRUE) can be achieved through conditional statements
In the envisioned (declarative) ODD domain specific language an ODD definition will be a collection of statements. This can be achieved by introducing to the language the following type of statement
propositional constraint statements
:
A statement specifying a propositional constraint, using a logical expression over previously defined variables,
which can be evaluated using "truth tables", i.e. each ODD sub-statement as well as any composite ODD statement can be evaluated to either TRUE or FALSE.
The semantics of propositional constraint statements define the mapping of every possible combination of values for its input expression statements to a True/False value.
Propositional constraint statements_statements may be defined over other constraint statements.
Example use cases:
A road is considered unsafe for the automated driving system, if there is heavy or medium rain.
Example representations:
-
Syntax 1
keep(rain not in [medium, heavy])
-
Syntax 2:
SUITABLE rain EXCEPT WHEN medium OR heavy
4.4.7. Operators
linked to requirement cluster [CL12]
The operators must match the syntax of the chosen language, for example the operators of range and others should match the encapsulation language
The following operators will be build into the OpenODD Format:
Operator |
Description |
Example |
Comment |
|
plus, Addition |
x+y |
will return the added value |
|
minus, Subtraction |
x-y |
will retrun the subtracted value |
|
multiplicator, Multiplication |
x*y |
will return the multiplicated value |
|
divider, Division |
x/y |
will return the divided value |
|
modulo, Modulus |
3%2 = 1 |
will return the modulo value |
|
Floor division |
x//y |
|
|
negation, equivalent to "NOT" |
!(y<5) |
|
|
asign |
x=y |
will assign a value to a variable |
|
equal |
x==y |
will return a bool |
|
not equal |
x!=y |
will return a bool |
|
equivalent "AND" |
x AND y |
will return true if x and y is true |
|
equivalent "OR" |
x OR y |
will return true if x or y is true |
|
less |
x<y |
will return a bool |
|
less equal |
x⇐y |
will return a bool |
|
greater |
x>y |
will return a bool |
|
greater equal |
x>=y |
will return a bool |
|
element in sequence |
x in [a,b,c] |
will return a false |
|
element not in sequence |
x not in [a,b,c] |
will return a true |
4.4.8. Hierarchy
The language format needs to be able to express the class hierarchy provided by the attribute group (WP01), it needs to support inheritance, be consistent with the class categories (keywords), data type and units. It shall support abstract ODD definitions by choosing different hierarchical levels.
Example use cases: Inheritance
Challenge: Define a restrictive ODD that specifies motorway as the allowed road type, and junctions are not suitable. In addition, everything else that is not mentioned is not allowed. In this case, when the hierarchical class 'Junctions' is used, the restriction will automatically inherited by all its sub-classes, this is similar to the next abstraction example.
Example representations:
-
Syntax 1
import PAS_1883 # Assuming this includes PAS1883 fully elaborated ODD extend junctions_type: # No junctions of any sub-type allowed keep(size == 0) restrictive_odd: ODD: keep(road_topology.drivable_area.road == motorway) # only motorways allowed keep(all_default_values()) # restrictive, only explicitly included values allowed
-
Syntax 2:
MODE Restrictive INCLUDE PAS_1883.owl SUITABLE motorway UNSUITABLE junction # this unsuitability is automatically inherited by all the types of junctions under the junction class
Example use cases: Abstraction
Abstraction allows the user to define the ODD at any level of abstraction corresponding to the Ontology
Define a ODD that allows all dynamic elements defined in the ontology, except when there is any type of rain. In this example, ODD can be specified at the 'Dynamic elements' level, the semantic echos with the inheritance property which means all the sub-classes of 'Dynamic elements' are also applied.
Example representations:
-
Syntax 1
import PAS_1883 # Assuming this includes PAS1883 fully elaborated ODD no_rain_odd: ODD keep(dynamic_elements.weather.rain == none) # Disallow rain
-
Syntax 2:
SUITABLE Dynamic Elements EXCEPT WHEN rain
4.4.9. Permissive and restrictive definition
Permissive/Restrictive modes can be considered in two levels — at the individual statement level, and at the global level.
At the individual statement level, the language should take a mixed permissive and restrictive approach within its statements in order to be flexible when defining what is inside/outside the ODD (using explicit statements for inclusion or exclusion along with conditionals). The user of the language is responsible when specifying multiple permissive/restrictive expressions to guard from changing scope.
At the global level, restrictive mode is recommended to be considered such that ODD attributes that are not defined with statements can be evaluated by default to NOT allowed. This is particularly handy in the usual case where your custom ODD definition is based on an existing ontology (see importing an existing ODD ontology at the beginning of your new ODD definition) or when expressing a narrow ODD. Below is a reference from MUSICC (Multi User Scenario Catalogue for Connected and Autonomous Vehicles) project which defines the global permissive/restrictive. Design considerations for ODD ontology by CATAPULT, March 2020. Users can choose to define an ODD using permissive at the global level where ODD attributes that are not defined with statements can be evaluated by default to allowed. This can be useful in expressing wide ODDs.
"Conditions may exist in the real world which are not modeled in the ODD ontology, and therefore cannot be represented in an ODD specification. With a restrictive ODD definition, it is clear that these are outside of the ODD, which is consistent with the SAE J3016 definition(since the ADS is unlikely to have been “specifically designed to function” with an unforeseen feature). If the ODD definition is permissive, it is unclear whether the element is allowed for the vehicle or not. […]For a hierarchical structure to be useful, it should be possible to use combined expressions (e.g.“Rural roads except motorways”); otherwise, all but the simplest of ODDs require every leaf item to be specified."
ODD statements' permissive/restrictive mode should not be confused with natural language permissive/restrictive classifications (i.e. in the natural legal language "The court allows act x" is classified as permissive language while in the ODD language the same expression denotes a restrictive mode). |
Example use cases: ODD definition for a T-Junction
For the example t-junction represented in the image and ontology below. Lets consider what permissive and restrictive means at a global level. For this we will have no 'local' statements so that we can visualize the effects of the 'global' permissive/restrictive.
Using a restrictive base state. A 'blank' ODD definition for the attached t-junction ontology would mean that all leaf attributes can be considered as outside of the ODD: restrictive - any attributes that are not mentioned in the ODD description but are present in the ontology (T-junction ontology) are assumed outside of the ODD’s scope.
Using a permissive base state. A 'blank' ODD definition for the attached t-junction ontology would mean that all leaf attributes can be considered as inside the ODD: permissive - any attributes that are not mentioned in the ODD description but are present in the ontology (T-junction ontology) are assumed inside of the ODD’s scope.
Note that in both examples ideally attributes that require numeric input would be separately defined, as these could not be simply included/excluded. i.e lane dimensions should be defined regardless of base state for clarity of description.
Example representations:
-
Syntax 1:
# PERMISSIVE MODE permissive_odd: ODD: keep(road_topology.properties.lane_dimensions in [3..5]m) keep(road_topology.properties.number_of_lanes in [1..2])
# RESTRICTIVE MODE restrictive_odd: ODD: keep(road_topology.drivable_area_elements.lane_elements. lane_divider.lane_marking == broken) keep(road_topology.drivable_area_elements.lane_elements.lanes == single_lane) keep(road_topology.properties.driving_direction == left_hand_traffic) keep(road_topology.properties.lane_dimensions in [3..5]m) keep(all_default_values()) # makes it restrictive
-
Syntax 2:
# PERMISSIVE MODE (declare a new element to be outside ODD) INCLUDE odd_ontology MODE Permissive SUITABLE laneDimensions is {min: 3.0, max: 5.0} SUITABLE numberOfLanes is {min: 1.0, max: 2.0}
# RESTRICTIVE MODE (declare a new element to be inside ODD) INCLUDE odd_ontology MODE Restrictive SUITABLE Lanemarking is ‘broken‘; SUITABLE Lanes is ‘single lanes‘; … UNSUITABLE driving_direction is ‘left_hand_traffic‘; SUITABLE laneDimensions is {min: 3.0, max: 5.0};
Permissive vs. Restrictive ODD
The table below illustrates the permissive and restrictive ODD at both the local (individual statement, i.e. syntax 1: keep(all_default_values()), syntax 2: SUITABLE, UNSUITABLE) level and global (overall permissive and restrictive mode) level. It can be seen that at the individual statement level, the element needs to be specifically listed, by assigning permissive or restrictive qualifier such element can be permitted or restricted. However, at the global level, the specific element does not need to be mentioned in the ODD description itself. The global level permissive or restive mode relies on the underlying ontology to work. If the element is present within the ontology but not listed in the ODD, a global 'permissive' or 'restrictive' mode will auto include or exclude such element. However, if the element is not contained within the ontology and is not mentioned in the ODD description, under 'restrictive' mode it will be excluded even if later on the ontology is extended to include such element. This is due to the fact that 'restrictive' mode restricts everything that is not mentioned in the ODD. On the other hand, under 'permissive' mode, if the element is not in the ontology then it will not be within the ODD either. But if later the ontology gets extended to include such element, then it will be inside ODD. This is due to the fact that under 'permissive' mode, anything that is in the ontology but not mentioned in the ODD will be allowed.
Element specifically listed? |
Restrictive |
Permissive |
Level |
yes |
Element is outside ODD (e.g, UNSUITABLE T-junction, keep(all_default_values())) |
Element is inside ODD (e.g, SUITABLE Motorway), keep(<attribute> == <condition>) |
individual statement |
No, but element is represented in the ontology |
Element is outside ODD |
Element is inside ODD |
global |
No, and element is not represented in the ontology |
outside the scope of ODD, even if the ontology is extended |
outside the scope of ODD, however, if the ontology is extended the element will be inside ODD) |
global |
Example:
A given ontology provides the definition of rainfall
as low
, medium
, heavy
, and thunder storm
.
Restrictive
low
rainfall
and medium
rainfall
is allowed
Permissive
cloud burst
rainfall
is not allowed
cloud burst
needs to be defined in the ontology
4.5. Extensibility
4.5.1. Extension by Function
For the evaluation of ODD boundaries and measures the extensibility by functions is necessary, these function will not be part of the ODD but can be reference in the ODD, to point the evaluation tooling to the correct function. Basic mathematical functions (like sqrt(), max(), min() ) or functions with respect to the application domain (e.g. distance() ) are required for conditions, limits and measures. They can come from supplied library or external binding (e.g. C, Python) For each function the input parameters and the return values must be defined
Example use cases: Bind function
binding function from an external library
Example representations:
-
Syntax 1
struct ODD: def speed_limit():speed is external.cpp("get_speed_limit", "libutils.so") speed: speed keep(speed < speed_limit() * 0.95)
-
Syntax 2:
INCLUDE get_speed_limit, libutils.so METRIC speed_limit_out IS speed_limit(get_speed_limit, libutils.so) UNIT 'm/s' TYPE float SUITABLE * WHEN speed < speed_limit() *0.95 # METRIC is a potential keyword for defining external functions and libraries, but will need further discussion within the group during the standardisation phase.
Example use cases: Binding a library
binding an external library and use a function from the bound library
Example representations:
-
Syntax 1
def weatherStatistics(where: list of string, when: list of string, query: string): real\ is external.cpp("weatherStatisticsQuery", "libDbApi.so") # this binds to external library set percentSnowFall: real = weatherStatistics(['Germany'],['2020'], 'frequencyPercent') # call to external library set max: real = 1 def percentageUnsuitableWeather() is (max - PercentSnowfall); # computation using the value returned from the query
-
Syntax 2:
INCLUDE weather_Statistics, libDbApi.so METRIC weather_Statistics IS external.cpp("weatherStatistics", "libDbApi.so") MEASURE PercentSnowfall TYPE float UNIT "%" RANGE [min - max] SOURCE ExtDataSource.query(weatherStatistics(['Germany'],['2020'], 'frequencyPercent', weather); FLOAT max = 1.0; FLOAT min = 1.0; METRIC PercentageUnsuitableWeather IS (max - PercentSnowfall) UNSUITABLE * WHEN (Percent_Snowfall = max) # METRIC is a potential keyword for defining external functions and libraries, but will need further discussion within the group during the standardisation phase.
4.5.2. Extensibility of ontology
Due to the diverse stakeholders and use cases, as well as an ever expanding ODD ontology, there are requirements to modify the underlying ontology from a base model. During sharing and exchange of ODD specification, such modifications need to be traceable within the ODD specification. The language therefore shall support the addition of new attributes and values that are not in the original ontology. It shall support the establishment of standard libraries, e.g. PAS 1883.
Example use cases: add an attribute:
Based on the current PAS 1883, user would like to add too_cold
attribute defined when the ambient temperature is below 20 degree C.
Additionally, user would like to add rain_light
attribute defined when rain rate is less than 5 mm/hr and droplet size less than 1mm.
too_cold
will be added onto the Environment
branch of PAS 1883, and rain_light
will be added onto the rainfall
branch.
Example syntax:
-
Syntax 1:
# adding another feature to the ODD environment type extend ODD_environment_weather_type: set too_cold: bool = (temperature < 20Celsius) # adding a feature to rainfall type extend ODD_rainfall_type: light_rain: bool keep(light_rain == (rainfall.rainfall_rate < 5mm_per_h and rainfall.droplet_size < 1mm))
-
Syntax 2:
INCLUDE BSI_PAS_1883.odd ADDCOND too_cold TO Environment/ODD ADDCOND rain_light TO rainfall/weather/ODD DETERMINE too_cold WHEN (ambient_temperature < -20) DETERMINE rain_light WHEN (rain_rate < 5) AND (droplet_size < 1) MEASURE ambient_temperature UNITS C MEASURE rain_rate UNITS mm/hr MEASURE droplet_size UNITS mm # Alternatively, the MEASURE could be combined into the DETERMINE statement to be more compact
4.5.3. Metrics
Detailed description of metrics can be found in the Metrics chapter
4.5.4. Probability and uncertainty
This is explained in more detail in the Syntax concepts for uncertainty and probability.
4.6. Integration with Scenario-based Testing Workflow
OpenODD language and format can be viewed as a stand alone language for several use case. In addition, there are many use cases where the format needs to easily be merged and combined with other component , in order to create a complete set of a test case ( or test cases ). When one looks at these test cases, the simulation and/or virtual testing world should be referred to. The next sections will describe several use cases, and the from these the requirements on the format/language will be derived. The key requirement is that that while OpenODD format is defining the ODD boundaries, it has to be integrated with the full testing system that interacts with the environment, and compares it to the ODD.
Several examples of use cases:
-
The same scenario ( expressed on OpenSCENARIO 2.0) need to be tested with different ODDs. This the scenario code itself, and the test specification need the ability to have a generic reference to an ODD, with assumed taxonomy structure
-
The scenario and test code need the ability to match current environment conditions to the on specified on the ODD.
-
The Scenario need to respond to events of environment changes, or ODD conditions changes.
-
The Scenario need to respond to HMI request and flags, and match them again current ODD definitions and current environment conditions.
-
The Scenario need the ability to observe actors, and match them against current ODD definitions in place.
Example use cases:
Define a ODD specification constraint using types and enumerations. The ODD specification shall use the same taxonomy as the scenario description.
Example representations:
-
Syntax 1
import odd_defs.sdl # the ODD type declarations/taxonomy # define odd_1 to be a subtype of any_odd, with extra constraints # restricting the permissible conditions struct odd_1: any_odd: keep(road_type in [motorway]) # odd_1 is not an entity that can be addressed in used in the scenario code
ODD expressed using OpenSCENARIO 2.0 Enumerated types and constraints: As the ODD taxonomy and definitions are actually expressed in OSC 2.0 language, any scenario code can be easily merged with these definitions. ODD values, struct members and other can be easily referenced and compiled together. Example: The ODD is defined as a struct entity, and can be used anywhere. ODD boundaries constraints are applied to struct members. -
Syntax 2
# The example below will corresponding to only the motorway from the odd_defs taxonomy is permitted for the ODD INCLUDE odd_defs.sdl MODE Restrictive SUITABLE Motorway
Compatibility with OpenX standards:
As mentioned above, OpenODD needs to be integrated with the full scenario based testing flow. In order to make this integration as easy and as smooth as possible, the following is expected:
-
Adherence to the same properties and entities in OpenX domain models and ontology.
-
Adherence to the same types, units and expressions syntax and semantics.
-
Adherence to the same conventions for variable names
-
It is also desired that the syntax will be the same – in order to enable re-use and integration
5. Attributes for ASAM OpenODD
5.1. Overview
OpenODD requires a set of standardized "ODD entities" that define and characterize the elements of an ODD and their structural relationships. Several ODD taxonomies already exist in the literature, and some of them have even been standardized to some extent, as BSI PAS 1883 or the AVSC Lexicon.
In the next sections, the following points are discussed: * present a list of requirements for OpenODD attributes * elaborate on current limitations of ODD entities modeling approaches in existing related standards * Propose, motivate and explain the ODD entities modeling approach that openODD takes * Present the list of ODD entities of choice for OpenODD and their relationships, making use of visual diagrams and referencing an interactive HTML file accompanying this concept paper, where the list, structure and metainformation about ODD entities of choice are present.
5.1.1. Requirements for OpenODD Attributes
The general requirements list for ASAM OpenODD includes the set of requirements which are specifically relevant to ODD entities. In addition, it is useful to highlight the following requirements related to ODD entities:
-
The ODD entities need to be hierarchical
-
The ODD entities need to be extensible
-
The ODD entities need to be aligned with other similar standardization activities, where possible
5.1.2. ODD Entities data modeling
Limitations of current ODD entities data modeling approaches
While containing a comprehensive set of relevant ODD entities/attributes, most of the available taxonomies have limitations that do not make them suitable to be used "as is" in a machine readable ODD definition format, such as OpenODD. One of the aforementioned limitations being the lack on a consistent ontological relationships between the ODD entities. For example, while most of the available ODD taxonomies are organized in a hierarchical fashion, often times the nature of the parent-child relationship is not semantically consistent throughout the whole hierarchy, and it is not being formally specified.
In order to cope with the limitations of existing ODD taxonomies, and develop a solution which is compatible with a formal and machine readable representation, we propose the solution detailed in the next section. The solution presented below doesn’t exhaustively capture all elements. Our goal is, instead, to detail a viable approach to model ODD entities and their relationship in a manner that is consistent, extensible and compatible with the other requirements of the OpenODD standard.
OpenODD ODD entities data modeling approach
We propose to organize OpenODD entities in two main taxonomies: ODD Attributes, and ODD Parameters. Each of the taxonomies is modeled (by definition) as a hierarchical tree where the semantics of parent-child relationship means "is a type of". Thus, the more in depth in the tree, the more specialized are the concepts with respect to the parent category. EXAMPLE: For example, we can refer to a simple tree like [Vehicle-→ Car -→ Sports car, Vehicle -→ truck -→ box truck] where every parent category is a superset of the child category, and, ideally, siblings nodes are disjoint sets. The parent-child relationships in our ODD entities are analogous to this example.
The taxonomy of ODD Attributes details out the actual "concepts" that are part of an ODD definition. ODD Attributes are of boolean data type, meaning that either they are, or they are not part of an ODD definition. EXAMPLE: Concepts such as "Environmental conditions", "rain", etc. are modeled in this taxonomy.
ODD Parameters define and quantify ODD Attributes, and are organized in their own tree. ODD Parameters data type may be integer, float, etc. and also include units of measurement (SI units like mm, mm/h, kg, etc), and potentially explicit value ranges. EXAMPLE: Example of ODD Parameters are "ODD Environment Parameter" "Rainfall rate", "droplet size", etc.
Both ODD Attributes and ODD Parameters can have quantifying discretization of child nodes.
ODD parameters are tied to ODD attribute and they define, enrich, augment, and describe the semantics of the attributes.
Please read hierarchy chapter
Linking ODD Attributes to ODD Parameters
OpenODD concept paper presents multiple (at least three) different format and syntax approaches that could be considered viable and fulfill the requirements for the upcoming standardization project.
Different options exist that can provide suitable implementations for defining the relationships between ODD attributes and ODD parameters, either leaning towards a more ontological approach, where such relationships are specified in the underlying ODD ontology, or a more programmatic approach, where such linkages are established through defined functions, classes, or procedures.
While section 4.3.8 of this paper already provides a first overview of syntax and format implementation of the linkage between parameters and attribute, a specific implementation choice will be made in accordance to the syntax and format approach adopted for the overall standard in the subsequent standardization project.
Extensibility of ODD Entities
Similarly to the linking mechanisms between ODD attributes and parameters, the extensibility of the ODD entities list (attributes and parameters) is a matter highly tied to the specific syntax and format of choice. Section 4.4.2 provides an overview of an approach to extend the underlying ODD Ontology with new attributes, parameters, their metrics, units, etc.
5.1.3. OpenODD Standard ODD Entities
OpenODD choice of ODD Entities is well aligned with BSI PAS 1883 standard, as well as the AVS Lexicon //TODO link resource The set of ODD entities proposed for OpenODD concept is the result of an in depth analysis of the aforementioned documents, which takes the entities listed in both, and organizes them in a comprehensive hierarchical structure.
5.2. Requirements for the attributes of ASAM OpenODD
-
Need to be hierarchical
-
Need to be extensible
-
Need to be compliant with existing standards (ASAM OpenX family and non-ASAM standards)
-
Need to be consistent with taxonomies used by ODD taxonomy standards.
-
Need to be parameterizable
5.3. Attributes for ASAM OpenODD
-
the goal is to provide a sample of the possible attributes
Attribute: Agent_Type
Attribute: Connectivity
Attribute: Illumination
Attribute: Particulates
Attribute: Weather
5.3.6. Parameters for ASAM OpenODD
6. Metrics
6.1. Understanding metrics
At first glance, ODD seem to be a merely qualitative list of objects and conditions that can exist in the specified working environment of an automated vehicle. However, the variety of use cases for ODD specification listed in this document suggests that in several project phases engineers want answers on many specific aspects of ODDs, or at least related to ODDs. These can include pure Boolean answers (e.g. Is a specific scene of a driving scenario inside vs. outside the ODD), answers on a qualitative scale (e.g. Is the present coverage of an ODD by a set of test scenarios poor, medium or good) or even quantitative answers (e.g. Coverage of equivalence classes of an ODD in percent). Just listing attributes and equivalence classes that define the ODD would not be sufficient in a safety and quality-oriented automotive development and release process. Therefore, the OpenODD language will have to provide means to define certain metrics (which does not mean that the OpenODD standard comes with an exhaustive list of such metrics; it is rather about enabling the language users to define them).
A definition of metrics for the purpose of the OpenODD language will be given later down. As an intuitive introduction, a metric is a function that maps some property about a thing (some quality attribute, for instance) or about several related things (some sort of a distance between them, for instance) onto an ordered scale (for instance the set of non-negative real numbers, or a point scheme reaching from 1 to 10 points). Metrics allow propositions about the quality, completeness, work progress etc. related to ODD specification or ODD usage (e.g. to derive vehicle requirements, to specify test or simulation scenarios). In some cases they allow comparisons between automated driving system with different ODDs (which AD feature allows the "bigger" ODD?) or between different test scenario sets aiming at covering a given ODD. It should be noted that ODD related metrics do not necessarily deal only with the ODD; they can involve propositions about a scenario or a set of scenarios, about performed test runs on a simulator or public road, about currently implemented vehicle features with respect to certain ODD conditions and many more.
Metrics can be defined on objects or situations, in particular on their various attributes by using logical or arithmetical operators (like AND, OR, +, -, greater than, less than) or mathematical functions (e.g. min(), max(), log() etc.). Some involve domain-specific functions (e.g. distance(), maxspeed() etc.) or queries to an ODD or scenario database for objects or conditions matching certain filters (e.g. select all roads in a given country with at least four lanes). Complex metrics can be calculated from other metrics that have been defined before.
6.2. Overview
This chapter elaborates requirements not only to the OpenODD language in general, but more specifically also to the language elements allowing the definition of attributes and equivalence classes of an ODD. The requirement to the language in general will be that it is able to host metrics definition and metrics usage as optional data elements associated with an ODD or objects or attributes thereof.
To host means that the grammar of the language has keywords (e.g. "define metric") and operators, e.g. mathematical operators, to define metrics and also to refer to existing metrics definition, i.e. to use them. A potential usage is to refer to an existing metric to define a new metric. The metric definitions made by the user can be stored in the same or an included OpenODD file. But we don’t mean by "hosting" that the OpenODD language does come with built-in metrics; metrics are in the language user’s responsibility. It can be decided whether or not the OpenODD language utilities will be shipped with some sort of a standard library that defines already some frequently used metrics |
Right below, a definition of metrics specifically for the purpose of ASAM OpenODD is given. Next, a couple of user stories for different roles in the development and validation process of automated driving functions are listed to better understand the use cases and requirements for OpenODD metrics. Note that the formal requirements, also with regards to metrics, are included in the global OpenODD requirements list. Finally, some examples are given how metrics could be calculated and how their definition and usage might look in different candidate syntaxes for the OpenODD language. In the stage of this concept paper we do not attempt to give a fully proven mathematical foundation, nor a final syntax for metrics.
6.2.1. Definition of Metrics in the Context of ASAM OpenODD
A metric is a function that transforms features or properties of an ODD - alone or in its relation to other ODDs, to scenarios etc. - into a number or an ordered set of qualifiers (e.g. good > medium > bad). It is not required that the function is mathematically defined and objective. Sources for a metric can also be expert judgement, as long as the metric is clearly declared in terms that the syntax is accessible to interpretation by a machine (e.g. in order to perform some comparison or statistics on it) and sufficiently reproducible in practice. However in most practical cases the metrics calculations will be based on a computing language, using machine readable data sources.
An example for the latter would be the severity ratings S0-S3 as defined in ISO 26262-3: They are not mathematically defined, but clearly described in terms of "injuries of AIS classification X are at least Y% probable". Experienced accident researchers usually come to very similar results for the same setting. However, the latter relaxation was just introduced in order not to exclude any human-centered or semi-quantitative metrics in cases where we don’t have any better option today - the main focus will be on metrics that can objectively computed from an ODD definition (and related formal descriptions, e.g. of scenarios). |
Notes:
-
The definition and interpretation of these numbers and qualifiers is user-specific and not necessarily accessible to an objective formal decision (e.g. „storm“ as the highest level of wind as opposed to „wind speed > 100 km/h“)
-
The underlying measures can be objectively physically measurable quantities, like lane width in m or rain intensity in mm/h. In the same way, physically objective ODD metrics can be formed, e.g. distance to leaving ODD in m.
-
By using metrics engineers can compare two ODDs, e.g. a system built for ODD2 can operate on 20% more German roads (or road kilometers) than a system built for ODD1, depending on how frequently certain ODD features statistically occur on German roads, or per road kilometer.
-
Metrics can relate scenarios, sets of scenarios or recorded road test data to an ODD, e.g. "What percentage (of km driven, or of time driven) of Scenario X is inside ODD Y?" or " What is the coverage of Scenario Set X with respect to ODD Y?".
-
ODD metrics can serve as foundation for metrics on scenarios and the quality of a Vehicle-Under-Test (VUT).
E.g. „In how many cases did the VUT correctly perceive the edge of an ODD before leaving of the ODD occurred?“
6.3. User stories for metrics of ASAM OpenODD
In this chapter the user stories for metrics are listed, these are based on the user stories listed in the beginning of this document. Using the mentioned user stories in this chapter regarding metrics, it is possible to derive the requirements for metrics. The user stories for metrics may differ from the user stories defined in Use cases for ASAM OpenODD, this is caused by the fact that metrics are an application using the ODD specification, while the user stories in Use cases for ASAM OpenODD, are focused on the format and language of OpenODD. It is important that the requirements from are metric side are known, when developing OpenODD, as the format / language needs to consider the use of metrics with the ODD specification.
6.3.1. Specification and/or quality engineer
Representativeness of ODD specification in comparison to real driving environment
As an ODD specification engineer, I want to measure the representativeness of my ODD specification to a real driving environment.
Example |
Approach |
Percentage of aspects allowed (or not explicitly forbidden) by an ODD specification w.r.t. a given ontology |
Use algorithms like calculation of an aspect-wise norm and IoU to determine the overlap of the addressed (i.e. explicitly allowed or explicitly forbidden) attributes in the ODD specification with all available dimensions and value ranges in the ODD. |
Percentage of aspects addressed by an ODD specification w.r.t. a (preprocessed / labeled) representation of relevant attributes in a set of driven scenarios or a statistical evaluation of traffic scenarios. |
Use algorithms like calculation of an aspect-wise norm and IoU to determine the overlap of the addressed (i.e. explicitly allowed or explicitly forbidden) attributes in the ODD specification with all available dimensions and value ranges in the ODD. |
Percentage of aspects allowed (or not explicitly forbidden) by an ODD specification w.r.t. a description of the requirements for a given target country or market (e.g. which road types exist in Germany, etc.) |
Use algorithms like calculation of an aspect-wise norm and IoU to determine the overlap of the addressed (i.e. explicitly allowed or explicitly forbidden) attributes in the ODD specification with data about real road traffic and environmental conditions, which can be retrieved from databases with public traffic statistics or preprocessed and labeled data from test rides. Optionally the metric can be weighted by the relative statistical importance of individual dimensions or values. |
Coverage of discrete values or value ranges of relevant attributes in a set of driven scenarios or a statistical evaluation of traffic scenarios by a given ODD. |
Use algorithms like IoU to calculate the overlap of a set of driven scenarios with the ODD specification and compare the actual occurrence of situations with the ODD specification. |
Statistical similarity of the probabilities assigned to single or combined values of different aspects with the corresponding statistical data from public sources or test rides (e.g. sum of squared errors). |
Preconditions: ODD specification is enriched with probability values for the situations or attribute values. Calculate a quality factor (e.g. sum of squared errors) that summarizes for all dimensions (single or combined) the statistical probabilities given by the ODD specification with collected data from test rides or public statistical material.. |
Quality and completeness of ODD specification
As an ODD specification engineer or a quality engineer, I want to measure the quality and completeness of my ODD specification to see what needs to be improved or when I‘m done.
Example |
Approach |
Absolute number of attributes addressed by the ODD specification. |
Sum all attributes that are mentioned (explicitly allowed or explicitly forbidden) by the ODD specification. |
Ratio of unaddressed (neither explicitly allowed or forbidden) attributes or attribute values / value ranges in an ODD specification and the total available set of values / value ranges defined by the underlying ontology or a golden sample ODD specification or a (preprocessed / labeled) representation of relevant attributes in a set of driven scenarios that characterize the target market. |
Number of all attributes addressed (i.e. explicitly allowed or explicitly forbidden) or unaddressed (neither allowed nor forbidden) in the ODD specification, divided by total number of attributes from reference source. Attributes can also be weighted by importance for the market acceptance etc." |
As the language is OpenODD by assumption, many quality attributes are fixed and don’t need to be measured (e.g. degree of formality, understandability etc.) - so completeness is the most relevant remaining quality property. |
Size of the ODD (i.e. how many environments are inside the ODD)
As an ADAS/AD designer, I want to measure the size and restrictiveness of my ODD or compare several ODDs by size or details to see how broad the usage scope of my ADAS is and how sure I can be about remaining risks.
Example |
Approach |
(Weighted) attribute values or value ranges that are forbidden (or not explicitly allowed, depending on specification style) by an ODD spec w.r.t. the ranges defined by the underlying ontology, or by an evaluation of actual driving scenarios, or by the regulations or situations in a given target country. + Particular example: What percentage of all road kilometers in Germany are suitable for a given automation feature according to its ODD specification (generally, or related to specific dimensions like road type, weather etc.)? |
Sum all attribute values (e.g. for road type: highway and country road) that are allowed by the ODD specification and divide them by all values that the ontology provides for that attribute (e.g. for road type: highway, country road, city street, off road). Optionally give them a weight according to their market importance (e.g. if the chances of snow in some target country are very low, give snow a lower weight in comparison to sunshine and rain). In a modified version this can also be applied to attributes with continuous value ranges (like lane width or visibility range). |
Comparison of two given ODD specifications by size (how much bigger is ODD one ODD in comparison to the other)?" |
Check wether one ODD1 is a subset of ODD2. Divide the norm (attribute-wise or aggregated to one weighted sum) of the smaller ODD by the norm of the bigger ODD." |
6.3.2. Data scientist
Coverage by the ODD or some ODD aspects by a scenario library
As a data scientist, given a scenario library and optionally a distribution of parameter values to concretize these scenarios, I need to provide a coverage metrics for the safety case.
Example |
Approach |
Coverage of discrete values or value ranges of relevant attributes in a set of driven scenarios or a statistical evaluation of traffic scenarios by a given ODD. |
Use algorithms like IoU to calculate the overlap of a set of driven scenarios with the ODD specification and compare the actual occurrence of situations with the ODD specification. Option A: Apply equivalence partitioning to all dimensions of the ODD that are not discrete by nature. All combinations of such equivalence classes form bins, of which the occurrences are collected. Monitor during scenario execution which parts of the scenario fall into which equivalence classes and count the occurrences for each bins. Define a minimum coverage (e.g. at least 10 occurrences per bin) and count for which percentage of bins the objective is already met by the set of scenarios. Option B: Assign to scenarios (which is an "infinitely thin line", not a space) a certain "width", i.e. a range of neighbor values per category that are considered as covered. E.g. driving with a speed of 50 kph could be considered as representative for every speed in the range 40..60 kph, likewise for weather conditions etc.. So every executed scenario is now a sub space of the ODD (or even the space beyond ODD limits). Form the union set of all executed scenarios, then calculate the IoU with the ODD. |
Exposure to scenarios or specific aspects of the ODD
As a data scientist, given an ODD, I need to determine the exposure to a specific scenario or to certain aspects of a scenario (e.g. certain weather conditions) specified in the ODD specification, so that I can determine the weight/contribution of the scenario results in the overall assessment.
Note: The term "exposure" can for instance be understood as "exposure" in ISO 26262-3 Annex B.3 in combination with the Note 1 to the definition of "exposure" in ISO 26262-1: "The parameter 'E' in hazard analysis and risk assessment represents the potential exposure to the operational situation." Example: "Medium Exposure Probability (E3)" corresponds to 1%…10% of average operation time of the vehicle.
Example |
Approach |
Exposure (relative portion in comparison to all driven hours or km) of values (discrete case) or value ranges (continuous case) per dimension |
Retrieve statistics of conditions in target country or aggregated and labeled results from many test rides and put into bins per value or equivalence class for each dimension of the ODD. Divide number of cases per total number of cases. |
Exposure for a combination of values (e.g. night and rain and highway and low traffic density) or a given scenario" |
value table or probability distribution -→ n/a for now (Metric in a broader sense)" |
6.3.3. Scenario Editor
Inclusion of a scenario within the ODD
As a scenario editor, given an ODD, I need to determine if a scenario lies entirely within the ODD.
Note: This is, at first, not a metric but a query, which is further explained in the section on ODD queries. But there are subsequent metrics, of which a few examples are given here.
Example |
Approach |
What percentage (of km, of time) of a scenario lies within the ODD? |
Form the intersection of the scenario and the ODD (note that one has the appearance of a |
What percentage of a scenario lies within the ODD with regard to a specific condition (e.g. lane width)? |
Same as 3., but decide the in/out question only by one specific dimension (attribute) of the ODD, e.g. the weather |
For how long (in time or in distance) did the ODD violation of a given scenario persist (generally, or with regard to specific attributes)?" |
Use intersection to filter out the parts of a scenario that lie outside the ODD and measure the time or distance driven of these scenario fragments, form maximum if needed." |
Determination of ODD boundaries (in general or related to specific ODD dimensions) by a scenario
As a scenario editor, given an ODD, I need to determine if
(a) a scenario induces a violation of ODD attribute boundaries,
(b) which attribute boundaries of the ODD may be violated by the scenario,
(c) by how much or
(d) how long they may be violated, which are important information when judging the fall-back or Minimal Risk Manoeuvre capability of an ADS.
Note: Cases (a) and (b) are queries (providing Boolean answer or selection or attributes or values for an attribute) and therefore discussed in the section on queries. Cases (c) and (d) are metrics (providing a real number or similar as an answer) and therefore discussed in the section on metrics.
Example |
Approach |
(c) Max value in centimeters by which the minimum lane width specified by the ODD spec was ever undercut. |
(c) For every part of the scenario that lies outside of the ODD because of a specified attribute (e.g. lane width) find the extreme value (i.e. min or max, whichever is further apart from the ODD definition boundary) and return this value in its natural unit (e.g. centimeters for lane width, mm per hour for rain strength etc.) - This metric works only for attributes with a continuous value range |
(d) Time in seconds for which we have been operating (or will be operating, if the scenario is just under definition) outside of the ODD (in general or related to one selected ODD attribute, like lane width)" |
(d) Determine from a recorded run or from a projection of the scenario under planning the point of time when the vehicle is leaving the ODD (in general or related to a selected attribute) and from there count the time until re-entrance into the ODD or end of the scenario or occurrence of a specified event (e.g. triggering of an emergency reaction by the vehicle under test)" |
6.3.4. Safety Engineer
Coverage of the ODD by a situation catalog for Hazard Analysis (Functional Safety, SOTIF)
As a safety (SOTIF) engineer, given an ODD and a Hazard Analysis (acc. ISO 26262 / ISO 21448), I need to determine the coverage of that situation set w.r.t. the ODD, or
Example |
Approach |
Weighted percentage of discrete values or value ranges of relevant attributes in the HARA situation set with respect to the same attributes in the ODD. |
Use algorithms like calculation of an aspect-wise norm and IoU to determine the overlap of the addressed (i.e. explicitly allowed or explicitly forbidden) attributes in the ODD specification with the list of situation attributes and the combinations thereof. Optionally the metric can be weighted by the relative statistical importance (exposure) of individual dimensions or values. It should also be checked that the situation catalog entries are actually used in the HARA. |
Exposure (occurrence probability) of scenarios or specific aspects of scenarios in the ODD
As a safety (SOTIF) engineer, given an ODD and a Hazard Analysis (acc. ISO 26262 / ISO 21448), I need to determine the exposure ("E-Factor") to certain situations (e.g. highway is 10% of average driving duration, emergency braking happens not more than 5 times per year on average etc.) based on the given ODD, so that I can classify the risk of a hazardous behavior in these situations during Hazard Analysis and subsequent SOTIF activities.
Note: The term "exposure" can for instance be understood as "exposure" in ISO 26262-3 Annex B.3 in combination with the Note 1 to the definition of "exposure" in ISO 26262-1: "The parameter 'E' in hazard analysis and risk assessment represents the potential exposure to the operational situation." Example: "Medium Exposure Probability (E3)" corresponds to 1%…10% of average operation time of the vehicle.
Example |
Approach |
Exposure (relative portion in comparison to all driven hours or km) of values (discrete case) or value ranges (continuous case) per dimension |
Retrieve statistics of conditions in target country or aggregated and labeled results from many test rides and put into bins per value or equivalence class for each dimension of the ODD. Divide number of cases per total number of cases. Optionally translate the percentages into the ISO 26262-3 scale (e.g. E4 = more than 10%, E3 = 1…10% etc.) |
Exposure for a combination of values (e.g. night and rain and highway and low traffic density) or a given scenario" |
value table or probability distribution → n/a for now (Metric in a broader sense) which allows calculation of combined situations (e.g. at night and at rain and on a highway). Note that in the general case the exposures to the individual dimensions are not stochastically independent. |
Coverage of the ODD by test scenarios as a means to demonstrate sufficient level of safety
As a safety (SOTIF) engineer, given a scenario library and an ODD, I need to determine suitable metrics for coverage of the ODD by validation scenarios, so that I can assess the progress of my safety validation and estimate the remaining risk due to unknown hazardous scenarios that still lie inside the boundaries of my ODD.
Example |
Approach |
(Weighted) Coverage Metrics for a Scenario Set vs. an ODD |
Form the intersection of the union set of all scenarios and the ODD. Note that a scenario has the appearance of a |
6.3.5. Development engineer
Performance test of runtime decision making about inside/outside ODD
As a development engineer I want to assess or test the performance of runtime decision algorithms that tell whether I’m about to leave the ODD, and to obtain a probabilistic estimate for the risk of a wrong decision.
Example |
Approach |
|
For 1.) and 2.): Record ground truth (inside or outside ODD) and output of the onboard decision algorithm over time for many test rides or simulation runs. Takes samples (e.g. at equidistant time intervals, or 5 seconds after entering or leaving the ODD) and divide number of |
Average / maximum detection time after leaving the ODD" |
Drive or simulate scenarios that contain a situation where the ODD is left. Record ground truth time instance for leaving the ODD and change in the output of the onboard decision algorithm and sample average time interval in between (special treatment for cases where leaving the ODD was not reported at all.)" |
Testing against the ODD specification
As a development engineer I want to check tests, scenarios, recordings, etc. against the ODD specification.
Example |
Approach |
How well does a given set of scenarios cover the ODD? (May need equivalence class partitioning and/or weighting)? |
Option A: Apply equivalence partitioning to all dimensions of the ODD that are not discrete by nature. All combinations of such equivalence classes form bins, of which the occurrences are collected. Monitor during scenario execution which parts of the scenario fall into which equivalence classes and count the occurrences for each bins. Define a minimum coverage (e.g. at least 10 occurrences per bin) and count for which percentage of bins the objective is already met by the set of scenarios. |
6.4. Concepts
Metrics on ODDs are an advanced topic (and therefore not part of the core language, but an extension). The definition and calculation of metrics makes use of many other (extension) concepts of the OpenODD language, and also the underlying ontology (e.g. defined in OpenXOntology language). In particular, concept of the following extension packages may be useful:
-
Math functions: As metrics produce numbers, there may obviously be a need to use mathematical functions that come from libraries (and that are provided by existing functions of a tool chain or written in a general purpose programming language like Python). Examples include sum(), average(), max(), min() or even more advanced statistical functions.
-
Data access functions: Many pieces of information needed to calculate a metric relate to the ODD, but are not part of it. For instance, if the question is what fraction of all usage times are suitable for a given AD function based on its requirements on visibility, then it will be necessary to consult an external database or file containing such information. This can be achieved by adapter functions that also come from extension libraries.
-
Partitioning: If not yet done on ontology level or when defining the ODD itself in OpenODD language, metrics may require some partitioning. For instance, if lane width is defined as a real number in meters, a metric may still what to calculate the fraction of narrow roads within the ODD, so a definition of narrow, medium and wide roads must be created. This can be done using the Partitioning and Equivalence Classes extension.
-
Exposure: Equivalence classes within the ODD can optionally be decorated with an exposure attribute (how frequently is this subclass of the ODD occurring on average driving, which is an important input to hazard analysis and risk assessment). Some metrics have for purpose to calculate such exposures from external statistics or scenario recordings, and use concepts from Exposure extension package.
-
Queries: It may be necessary to retrieve items of information on the ODD or external information sources like road maps, e.g. to find all roads of a certain road type.
Based on these concepts, new metrics can be declared by a given OpenODD syntax and keyword and can be defined using standard mathematical operators (+,-,\*,/) and the aforementioned mathematical functions like min(), max(), average() etc.
Detailed concepts how to implement the typical metrics examples from previous subsection will be elaborated in the standardization phase.
Some promising approaches examined so far are based on the interpretation of an ODD as a multi-dimensional vector of sets:
Imagine an ODD as a multi-dimensional set: E.g. if you have and underlying ontology defined by three enumerative (i.e. mappable to the set of natural numbers N) parameters, for instance
-
road type in terms of {highway, country road, city street}
-
the number of lanes (as a natural number)
-
weather in terms of {sun, rain, snow},
and two more parameters described by continuous values (mappable to the set of real numbers R), for instance
-
visibility distance in m and
-
lane width in m,
then any ODD on this ontology is defined as a subset of N x N x N x R x R, or if you want, as a 5-dimensional vector of subsets (in case of enums) or sub-ranges (in case of continuous values) of the individual sets that form the dimensions of the ontology.
A measure for the size of a subset for each dimension can be determined, e.g. the cardinality (number of elements) for enumerative attributes or the sub range as a fraction of the total range (e.g. ontology allows lane width between 2 m and 4 m, the ODD comprises anything greater or equal than 2.5 m, so the range within ODD is 75% of the total possible range for the dimension lane width).
Set operations like intersection or union can be formed on the sets that define each dimension of two given ODDs.
Combining set operations and size measures can render numerical weights for each dimension of the ODD.
If a single weighted number for the ODD as a whole (not just for each dimension separately) is required as a metric, weighting can be applied by component-wise multiplication with a weight-vector to provide a new, weighted vector.
If a ratio between measures is required, component-wise division can be applied to obtain a vector with numbers between 0 and 1.0 for each component. Again, weighting can be applied to consolidate these numbers to one single measure.
To compare two ODDs (and for many more purposes), the concept of the Jaccard Index or Intersection-over-Union (IoU) (see https://en.wikipedia.org/wiki/Jaccard_index), which is a similarity measure commonly used in object perception and other domains, can be adapted to ODDs.
A sketch of such an IoU adaptation could look like this:
similarityMetricVector(ODD1, ODD2) = componentWiseDiv(weightODDVector(myWeightsVector, intersect(ODD1, ODD2)), weightODDVector(myWeightsVector, union(ODD1, ODD2)); similarityMetric(ODD1, ODD2) = scalarProduct(componentWeightsVector, similarityMetricVector);
6.4.1. Some examples how ODD metrics could look like
The actual syntax for metrics and for the OpenODD language in general will only be defined as part of the standardization project, so it is not possible to provide working and syntactically valid examples at this stage. However, since the topic of metrics is quite abstract for many readers, some examples are given in a pseudo code style that might look similar in the final OpenODD language, but without anticipating design decisions about the final syntax at a later stage.
An example based on an ODD and external weather statistics
Setting: We want to find out what percentage of situations in some country (specifically related to the weather restrictions from ODD specification) are suitable to switch on automated mode for some vehicle. The ODD allows any weather but snowfall. We assume that a weather statistics database for selected countries and selected calendar years in the past is available and can be accessed via OpenODD extension library (we consider different programming languages for choice to implement this access function).
-
Sample Syntax 1
def weatherStatistics(where: list of string, when: list of string, query: string): real\ is external.cpp("wheatherStatisticsQuery", "libDbApi.so") set percentSnowFall: real = weatherStatistics(['Germany'],['2020'], 'frequencyPercent') set max: real = 1 def precentageUnsuitableWeather() is (max - PercentSnowfall);
-
Sample Syntax 2:
INCLUDE Math; # OpenODD Standard Lib, can also be a python lib INCLUDE Statistics; # OpenODD Standard Lib, can also be a python lib INCLUDE Units; # OpenODD Standard Lib, can also be a python lib INCLUDE Metrics; # OpenODD Extension Package that defines base class for metrics INCLUDE Queries; # OpenODD Extension Package that defines base class for queries on an ODD definition INCLUDE ExtDataSource; # OpenODD extension to access INCLUDE MyOntology; # This enables the use of all attributes within "MyOntology" incl. weather and roadtype SUITABLE [roadtype.highway roadtype.urban_roads roadtype.parking_lots] EXCEPT WHEN weather.snowfall; # Define a Metric: Percentage of time where the usage of my AD function is allowed in terms of weather conditions # Define reusable variables for min an max values. FLOAT max = 1.0; FLOAT min = 1.0; # Statistical Percentage where AD function cannot operate due to weather conditions MEASURE PercentSnowfall TYPE float UNIT "%" RANGE [min - max] SOURCE ExtDataSource.query(weatherStatistics(['Germany'],['2020'], 'frequencyPercent', weather); METRIC PercentageUnsuitableWeather IS (max - PercentSnowfall);
7. Uncertainty representation in ASAM OpenODD
7.1. Overview
To ensure that an autonomous vehicle is safe to both passengers and other traffic users, many regulatory environment require that 1) manufacturer provide a description of the ODD for this the vehicle is designed to operate, and 2) operators ensure to restrict operation to the ODD.
For details on what ODDs are and how they are defined, please refer to Understanding ODDs
7.1.1. Motivation
Excluding fuzzy and uncertain situations from the ODD restricts the availability of the autonomous vehicle. It is not practical to categorically exclude all situation in which a non-zero risk exists for unsafe outcomes. As an example, it is not practical to exclude freeways even though we know there is a low, but non-zero chance, to encounter people on the freeway, either due to accidents, or due to traffic jams, or some other reasons.
An ODD specification needs to determine in which situations is the vehicle required to drive safely. In other words, the ODD needs to decide, for each situation, whether it is excluded or not. The uncertainty working group is focused on making a black-and-white in-or-out determination when situations are fuzzy or otherwise are associated with uncertainty.
In the concept project the working group targeting the representation of uncertainty within the ASAM OpenODD language, established a list of requirements. These requirements do not contradict concept and requirements staten in the requirement clusters in Requirements for ASAM OpenODD. The requirements agreed upon are as follows:
-
Enable safety argumentation, acceptable risk analysis and HARA against ODD definition
-
Introduce statistical measures, chances of collisions, etc from available data.
-
It is complementary to safety and risk relevant standards
-
Support both quantitative and qualitative exposure specification, but exclude (for now) severity and controllability
-
Support definition of perception uncertainty requirements, sensor capability/limitations
-
Support definition acceptable risk against the ODD
-
Support definition of expected behavior of other road users, risk that pedestrians on motor way
-
Support references to external accident databases
-
Support various validation methods
7.2. Requirements for the uncertainty representation of ASAM OpenODD
-
The ODD must map every situation to an in or out of ODD binary classification
-
Represent Risk of Occurrences
-
Support discrete occurrences, for example pedestrians on the road
-
Support continuous occurrences, for example a stretch of icy road surface
-
-
Allow extending to represent other types of “newly discovered” uncertainties
-
Allow specification at multiple levels of details (LOD):
-
Level 0: No details, no uncertainty (baseline)
-
Level 1: Uncertainty of weather (need external data sources)
-
Level 2: Uncertainty of occurrence (need external data sources)
-
Level 3: …
-
-
Propositional (true/false) class attributes representing which is true in the worlds must be separate from measures.
-
Propositional (true/false) class attributes may be associated with qualifiers which are distinct from all other attributes and measures.
-
Allow ODD reference data originating from external data source.
-
Reference external data associated with weather occurrences
-
Reference external data associated with occurrences associated with traffic patterns and behaviors
-
7.3. Concepts
The following concepts need to be integrated as optional constructs using the extension mechanism defined in Extensibility.
The concepts in this chapter are explained using the example syntax2 and is explained in Introduction to syntax 2. Other syntaxes may use different keywords or constructs to illustrate the same concept.
7.3.1. Uncertainty Level of Detail
Statements in the language may be qualified or not. Non qualified statements are of lower level of detail than the qualified statements.
Data points necessary to determine if the situation/scene/scenario is within the ODD or not:
SUITABLE Freeway EXCEPT WHEN Pedestrian
SUITABLE Freeway EXCEPT WHEN Pedestrian@P8
The level of detail increases as we increase the number of qualified statements within the ODD specification.
The source of uncertainty information is outside of the ADS. The on board systems are not required to estimate the uncertainty. As such pedestrian@Q8 will not change its truth value frequently thus the ADS will not exit and re-enter the ODD due to sporadic detection of pedestrians. Therefore using qualified statements avoid flickering of the ODD status.
This enables different stakeholder to leverage different levels of detail of the ODD. For example, the regulators may use the lower level ODD (non-qualified) whereas the manufacturer may use a higher level of detail (qualified). |
7.3.2. Uncertainty Qualifiers Approach
Qualifiers represent qualitative probability in terms of upper bound on the occurrence rate. As an example, P8 is a qualifier. The proposition Pedestrians@P8 evaluates to "true" if the rate of pedestrian occurrence is lower than once in 10^8 hours. That occurrence rate is provided by sources external to the driving system. It is typically associated with a time frame, or a geographic region, or both. It is expected that this rate does not change frequently and is associated with relatively wide geographic regions.
Exposure can be bounded by qualifiers, because qualifiers represent an upper-bound of the probability of occurrence.
For example, restricting the ODD to Pedestrians@P8="false"
(e.g using SUITABLE * EXCEPT WHEN Pedestrians@P8
) implies that the ADS is allowed to operate either when pedestrians are not present (=probability of zero) or they are present with a rate lower than once in 10^8 hours.
The bound "present with a rate lower than once in 10^8 hours"
quantifies and controls the exposure.
Qualifiers are used to achieve True/False mapping of fuzzy uncertain situations, regardless of the root-cause of "fuzzyness" or uncertainty. Each proposition may optionally be qualified by multiple qualifiers.
A qualifier attaches the uncertainty or exposure value to the ODD attribute.
SUITABLE <condition> EXCEPT WHEN <condition>@<qualifier>
The ODD specification without qualifiers will be call "baseline ODD" and with qualifiers will be called "qualified ODD".
Example Syntax of Qualifiers
Qualifiers may be added to any proposition (not to a metric) as a comma separated list after @ symbol. For example, a qualifier called "P8" can be defined to evaluate to True only when the probability of pedestrians if <10^-8:
SUITABLE Freeway EXCEPT WHEN Pedestrians@P8
Multiple qualifiers can be added as well. Consider a qualifier C2 used to indicator that the confidence level in pedestrian detection is >99%. Both P8 and C2 can be specified as follows
SUITABLE Freeway EXCEPT WHEN Pedestrians@P8,C2
variable definition statement for qualifiers are as follows:
DETERMINE_QUALIFIER P8 WHEN occurrence_probability <10^-8" whereby the metric 'occurrence_probability' is a reserved keyword of which meaning is defined in the standard.
DETERMINE_QUALIFIER P8 MORE STRINGENT THAN P7 to specify relationship between qualifiers (see semantics below)
Semantics of Qualifiers
The qualifiers are relaxing the conditions accepting situations in which the main proposition would reject; they are negating the main proposition.
Without adding @P8, the above statements maps a situation to True if and only if Freeway=True and Pedestrians=False. With the addition of @P8, the above statements maps a situation to True if and only if freeway=True and (pedestrians=False or P8=True).
Adding multiple qualifiers accepts more situations. For example, without qualifiers, the statement "SUITABLE Freeway EXCEPT WHEN Pedestrians" accepts situation if and only if Freeway=True and Pedestrians=False. Adding @P8,C2 would accept situations if and only if freeway=True and (pedestrians=False or P8=True or C2=True).
Intuitively this makes sense:
-
When the probability of pedestrians is very low, the risk is acceptable, thus the additional 'or P8' accepts reasonable risk.
-
When the confidence of detection is very high, the risk is acceptable, thus the additional 'or C2' accepts reasonable risk.
The variable definition statement:
DETERMINE_QUALIFIER P8 MORE_STRINGENT_THAN P7
implies that:
-
when P8 evaluates to True, then P7 must also evaluate to True
-
When P7 evaluates to False, then P8 must also evaluate to False
Define qualifiers as: P8= prob <10^-8 P7= prob <10^-7 C2=99% confidence (1% error) C1=90% confidence (10% error)
The statement
SUITABLE Freeway EXCEPT WHEN Pedestrian@P8,C2
evaluates to True if and only if Freeway=True and (Pedestrian=False or P8=True or C2=True) implying that freeways are included unless there are no pedestrians with high probability or detection confidence is very hight.
The statement
SUITABLE Freeway EXCEPT WHEN Pedestrian@P7,C1
is less stringent and thus accepts more situations.
The concept of qualifiers is well defined for both environment and engineering metrics. Nevertheless, engineering concepts should not be included in ODD specifications. |
This approach implicitly assigns qualifiers in general, and P-values in particular, to constraints. For each constraints, all qualifiers are assigned a value, as part of the requirement to assign values to all variables. As such, semantically it is possible to group situations based on their associated qualifiers. For example, the following grouping is possible to identify rare situations of interest:
SELECT * FROM constraints WHERE ODD1 AND (Pedestrian OR P8)
Further, the following grouping is possible to partition situation according to their importance:
SELECT * FROM constraints WHERE ODD1 GROUP BY Pedestrian,Pi FOR i in 1..8
In other words:
-
Qualifiers extend the baseline ODD to include additional situation for which the risk is acceptable.
-
Multiple levels of stringency can be used to represent a corresponding appetite for risk.
An approach to define semantics for “fuzzy” situations. The goal of uncertainty representation is to determine in or out of ODD:
-
Convert an otherwise “fuzzy” or “uncertain” situation into a black-and-white true/false determination.
-
Qualifiers are the specification of the conversion rules.
-
Data supporting evaluation of qualifiers originates from specific environment conditions (not from sensors).
-
Qualifiers are optional at the discretion of the ODD author.
-
Adding Qualifiers Increases Level of Detail (LOD).
For representing uncertainty, there are two major options:
-
Pure True/False Approach: Definitions should not contain any uncertainty qualifiers, e.g.:
[green]#SUITABLE# Freeway [green]#EXCEPT WHEN# Pedestrian present OR IcyRoad
-
Qualified True/False: Increasing LOD using qualifiers, similar to refined road types, e.g.
[green]#SUITABLE# Freeway [green]#EXCEPT WHEN# Pedestrian@P8 OR IcyRoad@P3.
Generic Qualifiers Approach
-
Qualified True/False: Adding qualifiers provided as input, similar to road types, e.g.
[green]#SUITABLE# Freeway [green]#EXCEPT WHEN# Pedestrian@Q1,Q2,Q3 OR IcyRoad@Q4,Q5,Q6.
-
Qualifier Definition Statements, e.g.
DEFINE(QualifierInterpretation, P1,”Probability of Occurrence <10-1”) DEFINE(QualifierInterpretation, P2,”Probability of Occurrence <10-2”) DEFINE(QualifierRelation, P2, MoreStringent, P1)
Multiple Qualifiers
Uncertainty Level Of Detail (ULOD)
Example: How do we represent fuzzy “expectation”, e.g. likelihood of drivers running red light:
-
Driving culture has a major impact, e.g. India vs Italy vs Germany
-
Location has major impact, e.g. busy city with lights in sequence vs rural roads with few lights
-
Timing has major impact, e.g. after transition green to red.
-
Traffic conditions have major impact, e.g. during traffic jam.
7.3.3. Multiple Qualifiers Q1,…,Qn
Integrating Engineering Qualifiers
Approach supports Engineering Qualifiers, e.g. ML Confidence Values “Ci”:
-
Qualified True/False: Adding qualifiers provided as input, similar to road types, e.g.
[green]#SUITABLE# Freeway [green]#EXCEPT WHEN# Pedestrian@P1,C2 OR IcyRoad@P2,C1.
-
Qualifier Definition Statements, e.g.
DEFINE(QualifierInterpretation, C1,”Confidence >0.9”) DEFINE(QualifierInterpretation, C2,”Confidence >0.95”) DEFINE(QualifierRelation, C2, MoreStringent, C1)
-
SUITABLE Freeway EXCEPT WHEN Pedestrian@P8,C2 OR Cyclist@P5,C1 OR IcyRoad@P3,C2.
-
-
Language Requirements: Introduce a “DEFINE()” construct with a list of “relation types”. [_52]
8. Queries
While ODD definitions can be done by OpenODD files alone, queries and metrics on ODDs may sometimes relate only on information given in the OpenODD file alone, but sometimes also rely on external data sources (e.g. databases providing details on road types and geometries or weather conditions in a certain target country). While the technical access to external data sources like files and databases is performed via standard libraries, there must be a means to formulate queries like "select all roads with at least two lanes of at least 2.0 m lane width". This requires a query mechanism and an appropriate query language syntax (e.g. similar to the well-known SQL language). But also the use case of querying details on an ODD itself needs to be supported (e.g. for reporting or metrics), which can be achieved by the same query language.
8.1. User Stories for queries
8.1.1. Scenario Editor
Inclusion of a scenario within the ODD
As a scenario editor, given an ODD, I need to determine if a scenario lies entirely within the ODD.
Example |
Approach |
Given an ODD specification and a scenario (e.g. in OpenScenario language), check whether or not (Boolean answer) the scenario is entirely within the ODD boundaries. |
Form the intersection of the scenario and the ODD (note that one has the appearance of a |
Note: There are also related metrics in the context of these user stories, see section on metrics.
Partial inclusion of a scenario within the ODD
As a scenario editor, given an ODD, I need to determine if a scenario lies partly within the ODD.
Example |
Approach |
Given an ODD specification and a scenario (e.g. in OpenScenario language), check whether or not (Boolean answer) the scenario is entirely within the ODD boundaries. |
Form the intersection of the scenario and the ODD (note that one has the appearance of a |
Which part of the scenario lies inside the ODD? |
Similar to the example above, the part of the scenario that is not in the intersection is the part outside the ODD. |
Which particular attribute or condition of a (piece of a) scenario caused it to be outside of the ODD? |
Performing the intersection operation between ODD and scenario attribute-wise reveals which attribute(s) cause the ODD violation: It is the attribute(s) for which the intersection does not contain the entire scenario. |
Determination of ODD boundaries (in general or related to specific ODD dimensions) by a scenario
As a scenario editor, given an ODD, I need to determine if
(a) a scenario induces a violation of ODD attribute boundaries,
(b) which attribute boundaries of the ODD may be violated by the scenario,
(c) by how much or
(d) how long they may be violated#, which are important information when judging the fall-back or Minimal Risk Manoeuvre capability of an ADS.
Note: Cases (a) and (b) are queries (providing Boolean answer or selection or attributes or values for an attribute) and therefore discussed in the section on queries. Cases (c) and (d) are metrics (providing a real number or similar as an answer) and therefore discussed in the section on metrics.
Example |
Approach |
(a) Does a scenario include an ODD violation (i.e. at least a part of it lies outside the ODD)? |
(a) Form the intersection of the scenario and the ODD (note that one has the appearance of a |
(b) Which attribute violates the ODD limit, and over which time interval of the scenario? |
(b) FPerforming the intersection operation between ODD and scenario attribute-wise reveals which attribute(s) cause the ODD violation: It is the attribute(s) for which the intersection does not contain the entire scenario. The part of the scenario which is not in the intersection related to that attribute is the interval for which the boundaries are violated. |
8.1.2. Test engineer
Inclusion of a scenario within the ODD
As a test engineer, I would like to know that a scenario that I have created is within the ODD.
See identical user story for role Scenario Editor above.
Inclusion of given aspects from ontology / domain model within the ODD
As a test engineer, I would like to understand which element of my driving ontology/domain model is in or out of my ODD
Example |
Approach |
Given an ODD and the ontology on which the ODD is defined, which attribute ranges / choices of the ontology are inside or outside the ODD. |
Form the intersection of the whole ontology and the ODD. The part of the ontology that is in the intersection is part of the ODD. The part of the ontology that is not in the intersection is the part outside the ODD. |
Note: This query depends on the completeness of the ODD specification. If the ODD specification does not say anything about a certain attribute that is defined in the ontology (e.g. weather) then it depends on the overall assumption permissive vs. restrictive how this shall be interpreted. In case of permissive, the whole range given for this attribute by the ontology is inside the ODD; in case of restrictive, nothing of it is inside the ODD - which can make the ODD unusable in practice (e.g. vehicle cannot operate in automated mode under any weather conditions).
8.1.3. Data scientist
Coverage by the ODD or some ODD aspects by a scenario library
As a data scientist, given a scenario library and optionally a distribution of parameter values to concretize these scenarios, [green]#I need to determine
(a) which aspects of the ODD are covered, so that I can
(b) assess which scenarios need to be added to support effective ODD testing.#
The question about aspects is not unambiguous. It can address the questions, which attributes (that are known from the underlying ontology) are explicitly addressed by the ODD specification (see metrics on ODD completeness in Section 6), but also which values for each attributes that are covered by the scenario set, or even which value combinations are covered. And if attributes are not covered by a set of scenarios, this may have different reasons: For instance, if a certain type of simulation works with simplified geometric descriptions of the road network and does not simulate any light or weather conditions, we cannot expect it to address attributes like weather or light conditions from ODD specification, and there is basically no way to fix this. Other simulations may be able to simulate these attributes, but they have not been considered explicitly but clamped to fixed default values. Even the opposite case can happen, that a simulation addresses values that are not explicitly specified by the ODD specification: for instance, almost any sort of simulation will have to make an assumption on the lane width - but there are ODD specifications that do not mention the lane width at all. |
Even more complicated is the second question, which scenarios would need to be added to achieve completeness. It may be possible to point out combinations of attributes that still need to be covered, based on the given ODD, the existing scenario library and some completeness criteria. But even then it is not straightforward to (automatically) construct the most efficient selection of missing scenario so that the completeness goal will be achieved.
All of this is beyond the scope of the OpenODD language and therefore not elaborated here any further. The use cases given above for the role of the Scenario Editor may give some hints how to implement such queries (they be extended from a single scenario to a scenario library by forming the union set of all scenarios in the library.)
Note: In the same context, there are also related coverage metrics described in the section on metrics.
Inclusion of a scenario within the ODD
As a data scientist, given an ODD, I need to determine whether a specific scenario is included in the ODD, so that I can determine whether it needs to be included in the assessment of results.
See identical user story for role Scenario Editor above.
8.1.4. Inclusion of a specific event or condition occurrence within the ODD
As a data scientist, given an ODD, I need to determine whether an occurrence of something during a scenario is allowed by the ODD, so that I can determine whether the resulting vehicle behavior needs to be considered in the evaluation of correct / incorrect behavior (if a situation is in the ODD then we would expect the vehicle to handle it correctly, but otherwise the behavior of the vehicle is unspecified).
Note: The purpose is not to describe or monitor the vehicle’s intended or actual behavior; this would be beyond the scope of OpenODD. It is merely to check whether a certain occurrence in the scenario (in particular, in the vehicle environment) belongs to the ODD or not. If it does not, a potentially resulting misbehavior of the vehicle under test would not count as a failed test, because outside the ODD the vehicle’s behavior is unspecified.
Note: Currently there are no plans to make events or actions part of the OpenODD language. So only thing that can be detected in OpenODD is a condition in terms of whether attributes of a scenario take certain values outside the ODD. More advanced queries may be possible using other means, like queries on OpenScenario scenarios, considering an OpenODD file as a comparison object.
Example |
Approach |
Let there be a scenario from which the current values of certain attributes (e.g. visibility, ego vehicle speed) can be determined for each point in time and on ODD that contains restrictions on the same attributes. We want to know for a given point in time whether the specific attribute was inside the boundaries given by the ODD or not. |
Compare whether the current value of the attribute is inside or outside the range or choice defined by the ODD. |
8.1.5. Statistical distribution of parameters mentioned in the ODD specification
As a data scientist, given a scenario library, I need to determine (a) the statistical distribution of scenario parameters that occur in the ODD definition.
Note: This is going beyond the main scope of OpenODD. Only some outlines for a specific example can be given here.
Example |
Approach |
Assume that the ontology has some attribute with a discrete value domain (e.g. weather with values {sunshine, rain, snow, fog}), of which the ODD allows a subset (e.g. only sunshine, rain and snow, but not fog). Now it is interesting for several use cases to put relative probability tags to the values depending on their occurrence probabilities in a given target country, e.g. 50% sunshine, 30% rain, 10% snow, 20% outside ODD). |
Use extension mechanisms to access statistical weather data for the given country (statistics must be available and accessible on some database server or in a file, which can be accessed through OpenODD extension library functions; it must also present the values sorted by compatible equivalence classes as used in the ontology.) Get the percentages for the different types of weather and associate them with the values of the attribute. |
8.1.6. Safety Engineer
Coverage of the ODD by a situation catalog for Hazard Analysis (Functional Safety, SOTIF)
As a safety (SOTIF) engineer, given an ODD and a Hazard Analysis (acc. ISO 26262 / ISO 21448), I need to
(a) decide whether the the situation set underlying the Hazard Analysis covers the given ODD or
(b) list situations or attributes that are inside the ODD but not yet considered in the HARA situation set, so that I can assess the quality and completeness of the Hazard Analysis.
Note: This use case heavily depends on how the Hazard Analysis (and the underlying situation catalog) is represented in a specific tool, because there is no established standard exchange format for them. If the Hazard Analysis is performed using a plain EXCEL sheet, a custom solution would have to be scripted to retrieve and parse the strings for the situations and compare string-wise to the ODD / ontology entries. If a dedicated tool is used, it may provide an API that can be accessed through OpenODD extension library, or it may be able to work directly with OpenODD format and perform the query there. In the following only an outline for the proceeding can be given, assuming that the situation catalog of the Hazard Analysis can be retrieved in a format as if it was an ODD specification, and further assuming that both are defined over the same ontology.
Example |
Approach |
(a) Given an ODD and a situation catalog underlying a Hazard Analysis (containing discrete value choices for some or all of the discrete attributes appearing on the ODD specification, e.g. road types, and equivalence classes for some or all of the continuous-valued attributes appearing in the ODD specification, e.g. visibility). Are there any situations that are inside the ODD but not covered by the Hazard Analysis? |
Form a cut set between ODD and situation catalog. If the cut set is equal to the ODD then there are no forgotten situations. |
(b) Same as (a), but the question is: Which attributes and which attribute values for existing attributes are missing in the situation catalog? |
Compare the list of attributes. If there are more attributes in the ODD than in the situation catalog, these are missing. Then compare for each attribute the value sets (or form attribute-wise cut set). The values from ODD that do not appear in the cut set are missing. |
8.1.7. Infrastructure Operator
Vehicle warning via V2X when leaving the ODD (at vehicle operation time)
As an infrastructure operator, I want to have the ability to warn vehicles in advance if they are about to chose a route through my road network which most likely is exceeding their ODD
Note: This involves runtime decision making and V2X communication, which is outside the scope of OpenODD. Only an outline can be given how to detect whether a certain route is entirely within the ODD for a selected automated vehicle. It is assumed that the information about all relevant properties of a route (according to the attributes defined by the ontology, e.g. lane width or curvature or height of bridges can be retrieved from an external database via an extension library function and is in a compatible format that can be compared or intersected with the ODD (as shown above for the case of scenarios that are compared to an ODD); in particular, for each coordinate along the route all attributes according to the ontology can be retrieved.
Example |
Approach |
Given a route and an ODD specification of a certain vehicle: Does the route contain any portions that lie outside the ODD? |
Form the cut set of the route (defined by the attribute value for each coordinate point along the route) and the ODD. If the cut set is smaller than the entire route there must be ranges of the route that are not accessible for this vehicle based on its ODD. |
Navigation recommendations via V2X to choose route that does not violate the ODD (at vehicle operation time)
As an infrastructure operator, I want to give recommendations to automated vehicles which routes/roads/lanes to use depending on their ODD capabilities
Note: Giving routing recommendations is an advanced use cases going beyond the original scope of the OpenODD language and, thus, not outlined here. But the ideas from the use case above may serve as an inspiration how this task can be solved, provided that general knowledge on navigation and route-finding is available.
8.1.8. Development engineer
Runtime decision whether the vehicle is inside the ODD
As a development engineer, I need clear criteria how to decide at runtime whether I’m inside or outside of my ODD and to detect or predict that I am about to (unintentionally) leave the ODD so that I can implement detection and reaction functions for this case.
Note: The implementation of runtime decision algorithms is beyond the scope of OpenODD, but a typical use case based on an ODD specification in OpenODD language. In particular, runtime algorithms are bound to the restricted and sometimes inaccurate view of the environment through onboard sensors, self localization, HD maps and V2X, and the algorithms are limited by realtime constraints and onboard computational resources. Possible approaches include the detection or estimation of relevant attribute values of the ODD (e.g. lane width, weather) and/or looking up such properties from an enriched HD map that is available onboard, based on current location and prediction for the short-term future.
If it is for development and validation purposes sufficient to practice simulation of such scenarios on a host or a-posterior off-board evaluation of test rides with the actual vehicle, the same techniques as described above for the role Scenario Editor can be applied.
Testing against the ODD specification
As a development engineer I want to check tests, scenarios, recordings, etc. against the ODD specification.
Example |
Approach |
Is a scenario or recording from a test ride inside the ODD? |
See similar use case given above for the role Scenario Editor. |
What percentage is inside the ODD? (Note: This is rather a metric, not a query - see explanations in the subsection on metrics user stories) |
Form the intersection of the scenario and the ODD. Divide the size of the cut set by the norm of the entire scenario. |
Which parts of a scenario are outside the ODD? |
See similar use case given above for the role Scenario Editor. |
Which specific aspects cause it to be outside the ODD? |
See similar use case given above for the role Scenario Editor. |
9. Partitioning and equivalence classes
For many use cases it is mandatory to partition the ODD (as a whole or aspect wise) into equivalence classes. For instance, if lane width is a parameter of a road there could be three equivalence classes {narrow = below 2.0 m, regular = 2.0 … 3.0 m, wide = more than 3.0 m}. Equivalence classes are not necessarily formed on one single attribute, but sometimes also on combinations of several attributes (e.g. interstate highway means that road type is highway and lane width is at least 2.5 m) or conditional on other attributes (e.g. narrow is normally a lane with that is less than 2.0 m wide, but on a highway we say narrow even if the lane is less than 2.3 m wide). Such equivalence classes are a necessary precondition for several use cases around ODDs, e.g. some ODD metrics, in particular those who deal with the important aspect of coverage of the ODD by a set of scenarios. Sometimes, but not always, the partitioning is already made on the level of the ontology (e.g. if some ontology defines the strength of rain not in a unit like mm / h, but directly in equivalence classes like light rain, moderate rain, heavy rain). But this is not always the case and not always are the existing classes suitable for the purpose of some metrics, so this ODD extension provides means to create complex partitioning definitions on top.
Jason review comment: to me, the extension aspect of queries might only apply to external databases. query of the ODD itself is more of the toolchain or implementation tasks, rather than an extension of the language concept. The query properties of the ODD itself is already embedded within the WP2 language concept. |
10. Exposure and probability
Exposure refers to the uncertainty of an occurrence. Probability is the method of quantifying that uncertainty. "Probability of Exposure" refers to quantifying the uncertainty of the occurrence related to the exposure.
Not everything that can occur within an ODD is equally probable. For instance, foggy weather may be part of the ODD of some automated function, but is expected to be rare in comparison to sunshine, cloudy or rainy weather. It is not necessarily a part of the ODD specification to specify the exposure to certain situations or the probability of certain events to occur, but doing so can support many activities around safety and risk assessment of automated driving systems. For instance, the Hazard Identification acc. ISO 21448 requires to judge whether some risk is initially acceptable, which depends on the exposure to a situation in which the vehicle reaction can lead to an accident. The same considerations apply for the judgement of acceptability of the final risk, after all improvement and validation testing has been made. It is also a well-known strategy to assign different testing or simulation efforts to different scenarios depending on their probability to occur. Some considerations on this topic and the following one can be found in the section on Uncertainty.
11. Dynamic aspects
An ODD specification may tell that an ADS function is suitable on roads with a legal speed limits of 100 kph, but will not state how fast the ADS is allowed to drive at any particular instant (this belongs to the ADS function specification). But sometimes it may be necessary to refer to description of dynamic elements (typically other participants) in order to specify situations that are suitable or unsuitable for a certain ADS function. For instance, a city pilot may be unsuitable for streets with "frequent crossing of pedestrian". However we need to define what is "frequent crossing of pedestrians"? In case this is not included in the ontology, the ODD author may have to define this term. This entails that in some cases, language means to express the characteristics of road users' behavior.
For more details about the difference between ODD and scenarios please refer to Difference between ODD and Scenario.
12. Onboard decision making
An automated vehicle must be able to determine at runtime whether or not it is inside its ODD. If it is outside the ODD, safe operation is no longer guaranteed, because the AD function has not been designed and tested for this environment. Accordingly, the vehicle must then inhibit the activation of the automated driving function or, if the ODD is unintendedly and unexpectedly leaving the ODD while driving in automated mode, the automated function must be safely terminated. This can happen in many cases by alerting the human driver and handing over control, but in some cases (e.g. vehicle without a human driver, human driver fallen asleep etc.) there must be a "Minimal Risk Maneuver", e.g. to slow down the vehicle and pull over to the road shoulder.
To enable all of this it must be possible to determine (and even predict for the horizon of a couple of seconds or minutes) whether the car is inside the ODD and whether it is about to reach the ODD limits. This is challenging for many reasons:
-
The view of the car on its environment is limited by the performance of its onboard sensors and V2X capabilities and totally different from the "omniscient view" of an ODD specification author.
-
Algorithms must be fast (real-time requirement) and feasible with the limited on-board computing resources.
-
Some exceptional events (e.g. the detected presence of a pedestrian on a highway) may appear very suddenly and unexpectedly; furthermore, depending on the ODD definition, such events can be allowed inside the ODD as long as their statistical occurrence frequency remains under a certain bound (which is not trivial to decide from a car’s perspective, see topic Uncertainty).
To solve all of this is the AD function developers' task, and not in the scope of ASAM OpenODD.
However, the OpenODD language should provide extensions to support this job in the best possible way, for instance by supporting a formal notation of detection conditions that can serve to derive specifications for the runtime system or even code libraries that can be linked into the runtime software of the AD function, and also test cases to evaluate the performance of such functions in realistic scenarios. Also means how to annotate in enriched HD maps the fact that certain attribute values relevant to delimit the ODD are present or absent (e.g. certain lane width, certain traffic signages or V2X facilities etc.) can support runtime decision making.
This kind of support will have to be elaborated in the standardization phase and could be implemented by a specific add-on library or by general requirements to the ASAM OpenODD language.
13. Annex to ASAM OpenODD
13.1. Compatibility with OpenX Standards
13.1.1. Integration with Scenario-based Testing Workflow
Summary: OpenODD language and format can be viewed as a stand alone language for several use case. In addition, there are many use cases where the format needs to easily be merged and combined with other component , in order to create a complete set of a test case ( or test cases ). When one looks at these test cases, the simulation and/or virtual testing world should be referred to. The next sections will describe several use cases, and the from these the requirements on the format/language will be derived. The key requirement is that that while OpenODD format is defining the ODD boundaries, it has to be integrated with the full testing system that interacts with the environment, and compares it to the ODD. Several examples of use cases:
The same scenario ( expressed on OPENScenario 2.0) need to be tested with different ODDs. This the scenario code itself, and the test specification need the ability to have a generic reference to an ODD, with assumed taxonomy structure The scenario and test code need the ability to match current environment conditions to the on specified on the ODD. The Scenario need to respond to events of environment changes, or ODD conditions changes. The Scenario need to respond to HMI request and flags, and match them again current ODD definitions and current environment conditions. The Scenario need the ability to observe actors, and match them against current ODD definitions in place.
Example syntax:
ODD expressed using OPENScenario 2.0 Enumerated types and constraints: As the ODD taxonomy and definitions are actually expressed in OSC 2.0 language, any scenario code can be easily merged with these definitions. ODD values, struct members and other can eb easily referenced and compiled together.
Example:
The ODD is defined as a struct entity, and can be used anywhere. ODD boundaries constraints are applied to struct members.
import odd_defs.sdl # the ODD type declarations/taxonomy
# define odd_1 to be a subtype of any_odd, with extra constraints
# restricting the permissible conditions
struct odd_1: any_odd:
keep(road_type in [motorway])
odd_1 is not an entity that can be addressed in used in the scenario code.
13.2. Terminology of the uncertainty representation
"Uncertainty Types" relevant to ODD:
-
Uncertainty of Occurrences (past or future),
-
Uncertainty of Environment (fully or partially observed),
-
Belief in occurrence or environment
Term | Definition | Example Link |
---|---|---|
Probability of occurrence |
Given an "event generation process" and a binary classifier labeling an event as "1" for every occurrence,
it is the count of events labelled as |
|
Belief in occurrence |
It is an estimate of the "probability of occurrence"; The "probability of occurrence" is the "ground truth", and the "belief in occurrence" is the estimate. |
|
Qualitative belief in occurrence |
Whereas the true probability is always a numeric value, we can choose to represent belief of occurrence in terms of ranges For example, we can say that when the probability is <1% it is "low", and >99% it is "high". |
|
Extrinsic representation |
represent uncertainty in methods which are independent of data, e.g. mathematical expressions, formulas, etc. For example, an equation, .e.g gaussian distribution, or other symbolic representation not accompanied with data. |
|
Intrinsic representations |
represent uncertainty using the data, typically statistical methods, e.g. confidence intervals, explained variances, effect size, etc. |
|
data lineage |
traceability: the conduit through which data and its interpretation has passed; it is a chain of owners (or custody) and methods applied in each step |
|
source credibility |
the credibility of the source, as defined by the person using it |
|
data credibility |
the reliability of information source and the lineage |
|
numerical precision |
the level of detail in specification of a number, e.g. number of decimal points |
|
numerical error |
the numeric difference between a estimation and reality |
|
relative numerical error |
it is a ratio: "numerical error" divided by the number representing reality |
|
numerical accuracy |
a statistical statement on the error, e.g., confidence interval |
|
uncertainty in environment |
given measures/metrics representing the environment, it is a statistical statement on the error, e.g., confidence interval for each of these measures/metrics |
|
classification accuracy |
number of correct classifications divided by the total number of items classified i.e. number of (true positive + true negatives) divided by the total number of items |
|
false positives rate" |
well defined for binary classification only; the number of incorrectly classified "1" divided by the total number of "0" items per ground truth |
|
false negatives rate |
well defined for binary classification only; the number of incorrectly classified "0" divided by the total number of "1" items per ground truth |
|
precision, positive predicted value |
the number of correctly classified "1" divided by the total number of classified (=predicted) "1" |
|
recall, true positives rate, sensitivity |
well defined for binary classification only; the number of correctly classified "1" divided by the total number of "1" items per ground truth |
|
specificity, true negatives rate |
well defined for binary classification only; number of correctly classified "0" divided by the total number of "0" items per ground truth |
|
negative predicted value |
the number of correctly classified "0" divided by the total number of classified "0" |
|
false discovery rate |
the number of incorrectly classified "1" divided by the total number of classified "1" |
|
false omission rate |
the number of incorrectly classified "0" divided by the total number of classified "0" |
|
belief in safety |
the confidence in a statement "vehicle is safe to operate in an ODD" relies implicitly on defining an "unsafe occurrence" in the context of that ODD and estimating the "occurrence probability" of that unsafe event. |
|
scenario weight / probability of scenario |
When averaging results across multiple scenarios, we need to know the individual weights. Clearly, scenarios which occur more frequently should have more weight than rare scenarios. Assuming that the scenario is generated from a data driven approach, then |
|
crash probability for scenario |
|
|
probability of logical scenario |
This is the probability of an occurrence called "scenario", which requires identifying (=labeling) scenario from many driving hours across a large number of drivers.
|
13.3. Probability of Occurrence examples
13.3.1. The probability that a rolled dice will land on number "3"
-
Given a die and a method for rolling it
-
and given a method for labeling of "1" when the landed number is "3",
-
it is the number of "1" labels divided by the total number of rolls
13.3.2. The probability that a driver will run a stop sign
-
Given an intersection with a stop sign and a collection of vehicles passing through it
-
and given a logic for labeling of "1" when the vehicle is running a stop sign
-
it is the number of "1" labels divided by the total number of passing vehicles
Note that these examples assume ideal labeling methods. |
-
Accounting for imperfect labeling requires adding a conditional probability.
-
A full example of doing so is at the end of this doc.
13.3.3. probability of pedestrians on a road section" for a time period:
the "probability of occurrence" for events satisfying "pedestrian on road":
-
given a specific road section and a collection of vehicles passing through it for a given period of time,
-
and given a logic for labeling of "1" when "pedestrian is on the road" when the vehicle is passing
-
it is the number of "1" labels divided by the total number of passing vehicles
Rule for correct counting:
-
a vehicle passing a specific section multiple times is labeled and counted for each pass separately.
-
a vehicle passing multiple sections is labeled and counted separately for each section
-
aggregation over the time periods needs to be for similar time of day and other traffic characteristics, e.g. not mix 3am with rush hour traffic
-
weighted averaging should be done based on total count of events, i.e. vehicles passing, in each section, i.e. sections without passing vehicles should be excluded
-
"conservative assumption": the assumption that "the 1 labels occur independently" reduces the overall probability estimate as compared to situation where the labels are correlated
13.3.4. probability of pedestrians on a road section" for a time period for a specific ODD
Counting should only include events consistent with the ODD.
Comments and interpretation:
-
The "probability of pedestrians on freeway in Germany" is associated with a time frame implicitly requires aggregating across all freeways in Germany for that time frame.
-
When we don’t have perfect data, then we must estimate a number between 0 and 1 representing that probability, and hence we have a "belief that a pedestrian will be on the freeway in Germany"; the above concepts of numerical error and accuracy apply to that estimate.
-
Estimating probability of "pedestrians on A-9", for example, requires aggregating estimates across all its sections across multiple time periods.
-
If we know, for example, that we have a single pedestrian accident on A-9 freeway per year,
-
and we know the number of vehicles pass through the section of the accident annually, then we know the probability for that section is 1 divided by that number per year.
-
For all other sections of the road we get probability of zero. We can then average the results using event counts as weights.
-
-
For freeways which are not used the denominator (=count of events) is zero (no cars passing), and thus must be excluded from analysis.
-
The probability definition, and its estimate, requires counting, and thus it requires well defined counting rules.
-
Clearly, the count of both events and "1" labels over longer durations will be larger as compared to counts for shorter durations.
-
However, the ratio of the two count should change only slightly when that probability or its estimate is stable.
-
-
When in a traffic jam, drivers exiting their cars are pedestrians on the road, but HARA/HazOp should not label them as "1".
13.4. Uncertainty in environment examples
13.4.1. Example: one measure of "uncertainty with pedestrians in my environment":
given a subject vehicle and a pedestrian in its path, and given the measure of the distance of the pedestrian from the vehicle, it is the confidence interval of the distance estimation error
13.4.2. Temporal uncertainty
How temporal uncertainty should and can be represented in an ODD will be part of the standardization project.
13.4.3. Weather uncertainty
Given a proposition specifying a weather condition, such as "rain" or "icy road", we would like to specify the degree of uncertainty of the specific condition at each location.
Example: The forecast is for rain in downtown. There is uncertainty regarding the exact measure for each specific location. In one street we can have 10%-20% more precipitation than in other streets. Similarly, In one intersection we can have a 10%-20% larger (or smaller) average droplet sizes.
We need to have a definition of "rain" in terms of "average droplet size" and "average precipitation". We can define the proposition "rain" to be true if "average droplet size">min_value, and "average precipitation">min_value. We can also define the expected range of values for each.
The information about expected average must be provided from outside the vehicle, because the vehicle is only able to measure at specific points, and cannot determine the average.
13.4.4. multiple views of uncertainty
visually describe uncertainty by integrating multiple views or perspectives of the representation (extrinsic or intrinsic)
related terms:
-
"side-by-side view": comparing the same measure in multiple views side-by-side, typically visually
-
"coincident view", or "combined view": comparing the same measure in the same view, by transforming onto a combined view, typically visually in a single chart
An example of "coincident view", estimated distance of pedestrian from the vehicle:
d2 d1 d1 d2 . -----10-------20------ d1 d2 d2 d1
13.5. Open Issues
13.5.1. Format
t.b.d
13.5.2. Metrics
t.b.d
13.5.3. Uncertainty
The approach of simple propositional semantics in which propositions with optional qualifiers if very powerful: . It defines simple building blocks which can be assembled into very complex ODD specifications. . It supports multiple levels of abstractions, whereby the most abstract level includes only True/False propositions, such as 'Freeway' (easy to define for regulators), and additional definition expression increase the level of details. . Qualifiers further increase level of detail, and enable adding acceptable risk to the ODD thus providing support for safety engineering. . Qualifiers which represent environment uncertainty must be evaluated outside the vehicle and provided as input to the driving logic via V2X communication options. In contrast, engineering qualifiers not representing environment uncertainty can be evaluated on-board the vehicle. . Exchanging ODDs across manufacturers is possible by limiting the specification to the appropriate level of detail used in INCLUDE files. For example, different manufacturers may have difference C-values (confidence) in their testing while still conformant to the ODD specification which exclude those C-values. . Emergent situations are handled uniformly because the specification is "stateless" and does not include time. However, the language enables enrichment for engineering purposes, e.g. by adding behavior specifications, without adversely impacting conformance or exchangeablity.