Application Story

Situation

Automated driving (AD) functionality requires extensive software testing beyond what the automotive industry has typically conducted. As the industry strides toward deploying autonomous vehicles, statistical testing methodologies are important to ensure adequate test coverage over possible scenarios. The ASAM OpenSCENARIO DSL standard, an abstract scenario language, provides a powerful tool for conducting comprehensive and scalable simulation tests.

A leading automotive OEM company is developing a Level 4 AD system. Through its verification and validation (V&V) efforts, the team has created a set of requirements that govern the system’s AD functionality. The engineering team then creates simulation scenarios to test and validate each individual requirement.

To fully validate all of its requirements, the company needs to define a comprehensive set of tests to validate each version of its AD software in simulation. This can increase the AD system’s production timeline, potentially delaying time to market.

Success strategy

The automotive OEM needs to conduct testing using simulations that cover an expansive mapped region, primarily to reduce its reliance on costly and time-consuming on-road testing. Early on, the team decides to save time and resources by combining tests for requirement validation and validation of behavioral functionality into a single testing methodology.

To implement this type of dual-use testing, the company turns to Applied Intuition to implement ASAM OpenSCENARIO DSL. Using ASAM OpenSCENARIO DSL, the team writes scenarios in an abstract, high-level language. Writing these abstract scenarios requires the team to specify high-level constraints (e.g., desired actor behaviors, road conditions, and/or generic constraints on the scenarios’ map locations). Applied Intuition’s ADAS and AD development platform then utilizes these constraints to generate a thorough set of simulations based on the customer’s defined scenarios. Validation engineers can then control how many simulations to run for each abstract scenario using an intelligent queuing system provided by Applied Intuition.

After running the simulations, scenario engineers can decide which scenarios across the map are the most relevant for further optimization. The team can also run the same simulations directly in continuous integration (CI), running nightly on new builds of AD software and providing statistical pass/fail results for AD functionality across the mapped region.

Challenges during the project 

The core motivator for this project is to improve the automotive OEM’s V&V process by making the act of writing broad sets of scenarios simpler and faster. Most of the company’s developers are used to writing scenarios in a concrete or logical manner, where ego and actor maneuvers are declarative and rigid, and the location of a scenario on a given map is fixed. Once the AD simulation team identifies that writing scenarios in an abstract format will significantly improve its workstreams, the team begins to write new and migrate existing scenarios to an abstract language within Applied Intuition’s toolchain.

The team faces a key challenge when deciding on the abstract scenario language itself. Applied Intuition implemented its own abstract language prior to the release of ASAM OpenSCENARIO DSL, and the automotive OEM had been using it for two years. This means that key features of abstract languages, including automatically varying the map location and intelligently sampling parameter ranges, are already available within Applied Intuition’s toolchain. However, the ecosystem around ASAM OpenSCENARIO DSL is maturing. The OEM’s team identifies a few scenario vendors who will be able to provide ASAM OpenSCENARIO DSL scenarios. Applied Intuition then begins to migrate its abstract scenario editor UI, language, and abstract scenario building blocks (such as semantic map querying) to support ASAM OpenSCENARIO DSL.

Thanks to ASAM’s clear scenario language spec documentation and example scenarios, the team is able to easily support the ASAM OpenSCENARIO DSL abstract scenario spec while saving time and scenario engineering effort.

Business benefits

By using the ASAM OpenSCENARIO DSL abstract scenario standard, the automotive OEM is able to generate scenarios affordably across a large map, cover and test a broad range of different parameters in a scenario, and gain confidence in the breadth of its testing on a nightly cadence with CI-based abstract scenarios. Using a concrete scenario language could achieve similar results, but it would require the team to generate and maintain more scenarios by several orders of magnitude. Additionally, the growing ecosystem of vendors around ASAM OpenSCENARIO DSL enables the company to interchangeably purchase solutions that speak the same language, preventing vendor lock-in.