B.13 Workflows for test engineers

As a test engineer working for an AV/ADAS development company, I can define scenarios to build and run tests to test what other developers at other companies are running.

If you want to run a test using a scenario that was not build by yourself you need to import the scenario into your simulation platform somehow.

Usually another company (from now on called 'company B') delivers the scenario they use in some form.

Depending on the delivered format you need to follow different steps of conversion.

As a test engineer, I can define scenarios and run tests as close as possible to each other regardless of the execution platforms.

Simulation and test environments are offered from vendors and run on different hardware platforms. The creation of scenarios that are platform independent is key for a successful and reliable scenario exchange.

Creating scenarios and tests that can be used on different platforms is straight forward. Follow these instructions:

You now have a runnable scenario on your target platform with the same functionality as on your source platform.

As a test engineer/auditor/regulator, I can easily trace test cases, scenarios and results back to the requirements.

During a test campaign, OpenSCENARIO is just the tool used to describe scenarios and execute them in a simulator. At the same time, each one of these scenarios are linked to test cases and requirements.

All in all this use case describes how an OpenSCENARIO user is able to get the results of a test case execution within a simulation and trace it back to the requirements.

The person taking these steps is a test engineer that has to link results and requirements after simulator execution. Requirements, results and a scenario database already exist.

As a result all the requirements, test cases, scenarios and results are linked.

When the results are analyzed, tracing back the original requirements and the originating test cases can be done at any stage of the validation procedure.

As a test engineer, I can use OpenSCENARIO 2.0 to enable specification of a driving mission through inclusion of multiple maneuvers in a sequence or in parallel for both DUT and any other traffic agents.

OpenSCENARIO 2.0 scenarios may include composition and mixes of multiple actions and scenarios that involved with multiple maneuvers. This allows creating abstract building block libraries that are agnostic to the execution platform, ODD, automation function, maps and more. Within the behavioral block of a scenario, the user can trigger or observe other sub-scenarios to be create in parallel or serial ordering. These sub-scenarios may belong to multiple different actors, and their composition simplifies and reduce maintenance of complex scenario specifications.

In the example scenario C is created as a parallel composition of scenario A and scenario B.

The abstract definition in declarative format makes reading and writing of OpenSCENARIO 2.0 code easy. The use of abstract scenarios ensures an efficient automated process that meets your driving mission goals.

As a test engineer, I can use OpenSCENARIO 2.0 to accomplish a selected driving mission with an indication of whether the mission has been accomplished, what are the mission KPIs and how they are computed, and whether the unambiguous goals of the mission have been reached.

OpenSCENARIO2.0 scenarios are declarative and formal in nature. They allow the capturing of the high-level generalized intent using constraints. Therefore a tool can automatically plan multiple concrete instances of the generic scenario.

Once the scenario is executed, a tool may observe the actual execution. The tool can then report if the observed behavior meets the high-level intent. An OpenSCENARIO2.0 engine should be able to report whether the high-level intent was observed or not.

To accomplish driving missions, follow these steps:

With the observed data from this use case you get a clear indication if the driving mission or the scenario intent was achieved or not.

As a test engineer, I can create OpenSCENARIO 2.0 scenarios based on abstract test descriptions, for example textual regulations like NCAP tests, UNECE regulations or a requirement list.

The goal of this workflow is to have a scenario that can be used with any OpenSCENARIO 2.x compatible tool regardless of what kind of test description is given as input.

Test descriptions can be of different kind. For example, a NCAP test description, UNECE regulations or any other requirements list.

The expected output is always a compatible OpenSCENARIO 2.x file.

Follow these steps to transform an abstract test description into a scenario that is compatible with OpenSCENARIO 2.x.

The abstract test description is now available in an OpenSCENARIO 2.x compatible format.

As a test engineer, I can reuse/combine scenario elements and complete scenario definitions for the creation of new scenarios without the need of copy-paste.

Several parts of a scenario can be reused. In this context, it is appropriate to avoid copy-paste of the same parts all over several scenarios.

A much better approach is to store and reuse often used portions into libraries. This approach leads to a cleaner and simpler scenario. This approach also allows for easy corrections and sharing of parts.

As the need of a new scenario arises and its development starts, a good practice is to include a set of well-known libraries containing widely used and tested scenario elements and definitions.

When a given scenario element needs to be added to the scenario, one can search in the included libraries and reuse scenarios as-is without having to copy-paste it.

If such an element does not exist, the scenario developer has to create it and then he may add it to a library for reuse.

The same workflow can be followed for a given scenario definition.

In both cases the newly created scenario does not contain code that was copy-pasted. This increases the readability and the maintainability of the code.

As a result the new scenario contains smartly reused code instead of copy-pasted parts of already existing code.

As a test engineer, I have a clear guideline where to specify test aspects (what is inside an abstract or logical scenario and what is outside of it (for example, in experiments or test case definitions or in a map file).

OpenSCENARIO 2.0 as a standard is purposely created as an open and modular standard. This use case aims to shed some light into the different possibilities that a user has when using the standard so as to combine in-house or out-f-the-recommended tools and processes while still using OpenSCENARIO.

This use case defines a number of items that shall be part of the OpenSCENARIO 2.0 documentation as modularity is allowed by the standard.

OpenSCENARIO 2.0 is a modular standard. While there are preferred ways of using it and recommended ways of executing the scenarios that are defined by using it, some users or tool vendors may choose to deviate from this guidelines.

Because it is important that OpenSCENARIO is adopted by as many users as possible, it has been purposely defined as an open and modular standard, allowing all these modes of execution and without restricting its usage.

As a development project lead, I can create scenarios on an abstract level to document the functional behavior for legal reasons.

OpenSCENARIO 2.0 allows the capturing of the following references on any platform in a formal mathematical way:

This formal description, including the constraints and the coverage goals, provides a clear specification of the needs. This formal description leaves no space for interpretation.

Once you have captured the requirements and executed regressions, an OpenSCENARIO 2.0 tool can transform this accurate format into documentation and reports of both the scenarios and the thoroughness of the execution.

The results of this use case are accurate data-driven reports indicating both the goals and the achieved result in mathematical terms.

As a test engineer, I can use recorded real-world data to automatically detect/classify or generate scenarios that describe the recorded situation.

Real world recorded data can be used to further analyze a scenario. Raw recorded data can also be post-processed an re-played.

If a replay OpenSCENARIO 2.0 scenario file already exists, the recorded data can be re-played immediately. If such a scenario does not exist, it must be created before the recorded data can be re-played.

Starting with data recorded in a real-world situations, you have different ways to proceed:

As a test engineer, I can execute different OpenSCENARIO 2.0 files in an automated way with my OpenSCENARIO 2.0 compliant tool chain.
No file modifications are needed.

This use case shows the workflow that a test engineer must follow to use OpenSCENARIO 2.0 as a part of the execution process of tests within a simulator. This use case is oriented towards the tools vendors and how they may streamline the workflow of using OpenSCENARIO as part of their simulation tooling.

This workflow assumes that a database of scenarios exists and that the test engineer has been given some test criteria.

As a result of this use case it is proven that OpenSCENARIO 2.0 as a standard offers all the tools and capabilities to be used as a scenario description language that can be linked to success criteria for automated execution of test campaigns.

As a test engineer, I can execute the same OpenSCENARIO 2.0 files on different OpenSCENARIO 2.0 compliant tool chains.

OpenSCENARIO 2.0 (OpenSCENARIO 2.0) is a high-level language with a standardized syntax and semantic. This allows multiple vendors and tool developers to create OpenSCENARIO 2.0 compliant tools and leverage the same OpenSCENARIO 2.0 scenarios on different tool chains.

Note that different platforms may not support specific kinds of action. For example, a physical platform may not allow non-physical execution or an HMI that is not supported by the platform. The tool chain is still OpenSCENARIO 2 compliant in this case.

Running the same scenario on multiple platforms saves a lot of implementation time. It also improves the communication across teams as the teams share and exchange executable formats.

As a test engineer, I can leverage an OpenSCENARIO 2.0 engine to convert abstract scenarios into concrete scenarios which can then be tested.

Abstract scenarios include both free dimensions and ranges. Abstract scenarios also include dependencies between actions and actors. This allows technology to process and generate multiple legal concrete scenarios.

OpenSCENARIO 2.0 allows the capturing of abstract road descriptions. OpenSCENARIO 2.0 also allows technology to identify specific locations that meet for the abstract road location and the scenario needs. This provides a huge productivity boost for covering the entire ODD scenario space.

Note that a tool can provide a coverage result to indicate that the legal space of the abstract scenario was exercised.

The result of the automatic abstraction is a large amount of legal scenarios that deeply stress and validate the DUT.

As a test engineer, I can run the same test case in a MiL, SiL, HiL environment as well as in a real car controlled by driving robots or human drivers, with a real or partially simulated environment.

OpenSCENARIO 2.0 allows defining abstract scenarios that are agnostic to the execution platform. This allows mapping the platform agnostic scenario to MiL, SiL, HiL, proving grounds and so on.
Note that not all scenarios can be implemented in all platforms. For example, on SiL you may request a non-physical execution that is possible in a simulated world but cannot be achieved with physical testing like proving grounds.

OpenSCENARIO 2.0 allows creating reusable scenarios. You can also create a platform specific layer that notifies the user in case the scenario cannot be used on this platform.

Note that the implementation of specific OpenSCENARIO 2.0 actions might be different between vendors and platforms and is not part of the OpenSCENARIO 2.0 standard. The OpenSCENARIO 2.0 standard provides only the abstract standard definition and the desired execution semantic.

OpenSCENARIO 2.0 also allows external C++ code to be executed. This code qualifies the provided values for the specific platform.

Applying this use case results in a regression that is tunable for the desired platform.

Test engineers on a test track can define their own specific test track goals as coverage and work to achieve them.
They can also leverage the same scenario with other platforms.
In such a way tolerances and deviations between test description and test execution become obvious.

The same OpenSCENARIO 2.0 scenarios can be used in all execution platforms including test track.

Typically, SiL regression goals are more thorough than test track goals. Some requirements may only be verified with physical testing.

OpenSCENARIO 2.0 platform agnostic scenarios allow the setting of different goals corresponding to the platform strengths and capabilities. The scenarios can be generated by either team or adopted from a reusable library.

Test engineers can capture the desired scenario in OpenSCENARIO 2.0 formal executable format to be leveraged in other platforms.

The language extension capabilities allow the adjusting of the following settings:

Note that it is also possible to adopt scenarios that were created for SiL purposes and run these on test tracks.

As a result test engineers leverage their knowledge of the test track. The gathered knowledge can be used early on for SiL or other platform regressions.

As a test engineer, I can set goals and measure the V&V progress using functional coverage.

OpenSCENARIO 2.0 enables automated translation of abstract scenarios into multiple legal concrete scenarios.

The automated concretization process allows a tool to select random values for the missing abstract scenario attributes that are consistent with the rest of the scenario.

To ensure that all the desired scenario goals are achieved, OpenSCENARIO 2.0 provides a rich set of functional coverage features to capture the project goals. This executable forma; plan allows a user to measure the V&V status throughout the verification process and improve the quality of the DUT.

The following steps introduce an end-to-end V&V process with the special focus on the coverage related aspects.

The following code demonstrates the standard way for setting functional coverage goals.

Note that you may have different goals for different execution platforms. As a methodology, it is desired to place the coverage definitions in a file extensions and to select the right extension for proper platforms.