Development Update OpenSCENARIO V2.0

ASAM OpenSCENARIO 2.0 - Development Update

Status Nov 06, 2020

Status: Nov 06, 2020

1. Introduction

The AV industry moves at a very high pace, one that often far outstrips the usual speed of standards development. For this reason, the ASAM OpenX projects have been generally been kept to 1 year development pipelines, leading to a more rapid release cadence that is better aligned with industry.

At ASAM we felt that even a time frame of 1 year might not be sufficient to keep up with the pace and so decided that even if the OpenX project work is restricted to ASAM members, we need to keep the industry up to speed. This is the first of multiple quarterly updates on the ASAM OpenSCENARIO 2.0 standard. With it we want to provide insight into the development direction of the format to give the industry as a whole a chance to align with, provide feedback on or participate in the further activity for OpenSCENARIO 2.0.

2. Project Overview

The OpenScenario 2.0 standardization project was launched in June. It is split into 5 groups, each of which consists of multiple work packages (WPs) targeting specific work items, see the proposal for details.

Figure 1. OpenSCENARIO 2.0 Project Groups

Despite the current situation with work from home and no face to face meetings, all 5 groups are making progress towards delivering a first draft of the standard in Q3 of 2021. The teams are expanding and leveraging the concepts developed in the concept document.

Although OpenSCENARIO 2.0 is defining a domain specific language as a significant superset of OpenSCENARIO 1.x, particular emphasis will be placed on identifying migration rules from OSC 1.x to 2.0.

Disclaimer

The content shared in this update stems from ongoing development work and is thus subject to change without notice!

3. Project Timeline

The original project schedule was defined very aggressively, with the project opting to focus on laying solid foundations for continued development projects and also allowing for realignment with the industry. This is a similar approach to that of the OpenSCENARIO V1.x development path.

A full project meeting was held at the end of September where all groups presented their status. In general, all groups are in or nearing the norming & performing stages (see here) of their development work.

4. Group Status Updates

4.1. Usage & Pragmatics

This group aims to understand the usage of scenarios and the corresponding workflows that would make use of OpenSCENARIO 2.0. It will release documented workflows and accompanying guidelines on using the standard in them as well as guidelines on the different levels of scenario abstraction.

The Glossary created from the Concept stage has been reviewed and finalised.

The group also reviewed and updated the definition of the use cases from the Concept project and have decided on 42 unique Use Cases across 9 categories.

Use Cases are assigned to team members who are currently modeling the workflows for these. The group is aiming to investigate the possibility of having interactive, combined diagrams together with the service provider. One such example is demonstrated below.

Figure 3. As a tool developer, I can reutilize constructs, artefacts and libraries to create tools compatible with other tool vendors in industry (Concept paper S1.3.3.1.1)

4.2. Language Concepts

This is the most modular of the OpenSCENARIO groups, tasked with the design of the DSL. Many WPs have been making good progress on various design decisions:

Further foundations for the DSL have been laid, building on top of those developed in the concept project. The first drafts of the basic and compound types were completed. The WP on expressions has been along similar lines, defining physical units and basic operators, now looking at atomic expressions and typecasting.

The work on the surface syntax has not started yet as this is not as critical when some more fundamental semantic & pragmatics topics are still open. This will likely begin early 2021.

The WP defining libraries has started working on namespacing and fallback mechanisms. This WP will also be closely aligned with the OpenLabel project group with respect to scenario meta data.

The Semantics working group are currently focussed on putting together an iterative process to formalise the semantics for OSC 2.0. They have proposed a layering of the OSC 2.0 concepts which increase the power, flexibility and complexity of the language (with the final layer being the full language). The next few meetings will be used to describe the semantics for the basic layers.

4.3. Domain Model Group

The Domain Model group has established a goal to create a deliverable that is easily readable for new and veteran users of OpenSCENARIO. The domain model will provide a clear view of the scope of the language and a basic representation of its structure. Furthermore, the domain model will be used to ensure that OpenSCENARIO 2.0 is effectively a superset of the functionalities of OpenSCENARIO 1.0.

During the first quarter, the group reviewed relevant documentation from other bodies (such as BSI PAS 1883 (operational design domain), ISO 345xx (ISO/TC 22/SC 33/WG 9), ISO 8855 (coordinate systems), DIN Society of Automotive EngineersSAE SPEC 91381), which will help provide content and clarify the scope of the domain in OpenSCENARIO 2.0.

Sources of Information:

OpenSCENARIO 1.0
Open Simulation Interface
OpenXOntology
CARLA
M-SDL
BSI PAS 1883
ISO 3450x
stiEF
OpenLABEL

In the next quarter the group will build the first draft of the domain model and identify areas where ontological definitions can enrich the standard.

Domain Model content categories

Vehicles
Vehicle Actions
Pedestrians
Pedestrian Actions
Environmental Conditions
Coordinate Systems
Road Abstractions
Traffic Signals & Communication

Some early examples of domain entities and attributes can be found in the concept paper

4.4. Measurement & Success Criteria

The measurement and success criteria group made good progress on both fronts and was able to reach consensus through meaningful discussions.

4.4.1. Success Criteria

Much progress was made on both syntax and methodology levels.

There was a large amount of discussion here as well as in other ASAM projects on understanding the interaction between a test and a scenario. ASAM has initiated a Test Specification Study Group to more closely investigate this interaction from a testing perspective (See proposal / Signup for the kickoff on 17.11).

The OpenSCENARIO 2.0 project group has also defined a task force prior to the aforementioned study group. Their goal is to begin understanding certain definitions that are important for the language development

So far, the group has established some base definitions.
(Note: The information below and any further detail is still subject to group discussion and hence subject to change!)

Test Case	Test cases can also be carried out without need for a scenario (requirement-based testing). Test cases may also be applied to other areas The Test Specification Study Group aims to define and standardize the definition of a test case in a more general picture englobing the automotive world use cases
Scenario	A scene in a 3D world in which actors are allowed to move. A scenario shall contain all the necessary information that describes all static and dynamic entities in a specific scene The static entities contain but are not limited to: roads and furniture, buildings, nature elements and static actors. The dynamic entities contain but are not limited to: actors (vehicles, humans, …), road furniture (dynamic signs, traffic lights, …), road modifications (barriers, construction works) and weather/environmental conditions A scenario can be parsed and executed by a scenario engine

Test Case

Test cases can also be carried out without need for a scenario (requirement-based testing).
Test cases may also be applied to other areas
- The Test Specification Study Group aims to define and standardize the definition of a test case in a more general picture englobing the automotive world use cases

Scenario

A scene in a 3D world in which actors are allowed to move.
A scenario shall contain all the necessary information that describes all static and dynamic entities in a specific scene
- The static entities contain but are not limited to: roads and furniture, buildings, nature elements and static actors.
- The dynamic entities contain but are not limited to: actors (vehicles, humans, …), road furniture (dynamic signs, traffic lights, …), road modifications (barriers, construction works) and weather/environmental conditions
A scenario can be parsed and executed by a scenario engine

For OpenSCENARIO 2.0 as a scenario description format, some aspects of the interaction between these terms need to be clear, particularly the question of what types of success criteria are used where. Currently the group is considering two types of success criteria:

Test case success criteria that tell the user whether the ego vehicle and its functions performed as expected or not
The group identifies three sub-levels of criteria:
- Global checks – e.g. make sure that there is no collisions at all times
- Reusable scenario checks – e.g. in a specific scenario, allow the TTC to be 3s and nowhere else
- Test level checks and project specific refinements – e.g. with project x the TTC of 3s is not enough,
set it to 4s, with only a warning for a 3s TTC.
Optional scenario validity criteria, that allow one to check whether the scenario was executed according to expectations
E.g. - Vehicles changing lane to stay on the right may lead to scenario not generating the "dangerous" or "interesting" situation intended

Further refinements or disabling certain checks can be specified with special syntax in order to enable both reuse and flexibility.

Information

OpenSCENARIO 2.0 will not prescribe where and how to place success criteria, but it will provide multiple example workflows that demonstrate the various approaches. OpenSCENARIO 2.0 as a language will provide constructs that allow a user to define these as needed. Clarifying these approaches further, or even providing a recommendation on what should be where, will be part of the scope of the Test Specification Study Group. Interfacing to external test case specifications will however, be included in the language.

4.4.2. Coverage

The definition of coverage from the concept project, especially that of functional coverage, was reviewed and refined. The reuse and extendability aspects of coverage were given particular focus.

Figure 5. Reusable coverage definition can be targeted and refined for multiple execution platforms

Coverage in OSC 2.0 allows for evaluation of the proportion of the scenario space in which a DUT has been tested in a manner that is agnostic of the execution platform.

The proposed coverage features allow for capturing a wide range of goals of any expression (discrete or continuous) within any OSC2.0 container (actor, action, scenario). An associated sampling event allows for refined control of the sampling time at any desired phase or throughout the execution.

Much time was spent on how to capture white box coverage and the usage of external methods to target the generic code for specific execution platforms.

While coverage can be defined on any container, we review the right methodology to maximize reuse while reviewing several usage examples.

Figure 6. Example of the proposed coverage syntax

Information

The exact syntax for defining coverage is still undefined, as the group is waiting for the syntactical building blocks to be defined by the other language working packages.

Special syntax is proposed to override generic coverage definitions in a non-intrusive way to enable project and ODD customization.

4.5. Feature Sets

The subdivision of OpenSCENARIO 2.0 into feature sets was proposed to allow easier support and usage of the language.

One major drawback of complex formats or languages like OpenSCENARIO 1.0 is that tools or frameworks will not be able support the complete content from the start. This leads to a limited usability of the standard for exchange between different parties. Supporting frameworks have to provide compatibility guidelines with lists of format elements and their individual level of support (see e.g. esmini).

OpenSCENARIO 2.0 attempts to solve this through the definition of feature sets. Supporting tools and frameworks can indicate partial support for the standard using clearly delineated, standardized feature sets. This reduces the effort spent on accompanying compatibility documentation and additionally allows a quick identification of suitable tools for potential users of the standard.

Current Feature Sets (subject to change)

Replay	Concrete trajectory based scenarios Well defined actor behavior without interdependencies Used for recorded scenarios
Reactive	Complex synchronization between actors Introducing parallel as a composition operator Used for Human-in-the-loop use cases
OpenSCENARIO 1.x	Fully runtime compatible conversion of scenarios from OpenSCENARIO 1.x
Test Definition	Elements for measurement and success criteria Definition of KPIs Parameter value distributions Used to separate scenario and test definition
Basic Declarative	Simple vehicle behavior like lane change with parameters Sequential combination of multiple phases Used for simple logical scenarios like Cut-In

Next Steps

Until early 2021: Continue developing the individual feature set proposals, selecting the relevant domain model entities and language elements for each set.

5. The OpenSCENARIO Implementers Forum

In parallel to the development of OSC 2.0 ASAM aims to initiate an implementer forum, a platform for tool vendors, OEMs and other volunteers. Ideally the first step would be to agree on a set of benchmark/reference scenarios and describe these based on individual understanding of the current draft of the OSC 2.0 language. This will serve to help identify gaps or issues in the specification that could otherwise not be identified until after release of OSC 2.0. These descriptions should adapt and grow in parallel with the developments to the specification.

The Implementers Forums will also ensure that a wide range of companies in the industry already have a good understanding of the usage and implementation requirements of the standard by the time of its first release. The implementers forum will continue independent of the end of the first project. Once a full specification has been released, the participants of the forum will have a platform to continue exchange and compare their growing implementations of the standard (through e.g. cross-tests). This platform also provides a link to OSC 1.x, allowing for demonstration & testing of migration between 1.x & 2.0.

Note:

If you are interested in participating in this group (regardless if you are already involved in the development project or not), please get in touch with benjamin.engel(at)asam.net. We are also still on the lookout for a lead for this forum.

6. Closing Words

The current difficulties due to COVID19 faced by everyone around the globe have of course also presented a hurdle. This project consists of over 100 participants, where many have never met face to face or have never worked in an ASAM project. The fact that this group of members, with a truly worldwide representation, have found a productive working mode, are making progress and are still on track for the planned release, speaks volumes about their engagement not to mention the demand in the industry for a standard like OpenSCENARIO 2.0.

I look forward to our continued virtual discussions and very much hope to be able to meet all the contributors in person in the not so distant future!

If there any questions or comments, please feel free to contact benjamin.engel(at)asam.net.