B.8 Workflows for safety engineers
For safety engineers the study of the following workflows is recommended.
B.8.1 Workflow list
-
Section B.8.2
As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to derive a simulation scenario from findings of a SOTIF analysis. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction. -
Section B.8.3
As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to derive new hazardous events, system insufficiencies or triggering conditions for SOTIF from findings during a simulation run. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction. -
Section B.8.4
As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to replay different variants of a critical scenario observed on the road in simulation, to derive new SOTIF insights from it. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction. -
Section B.8.5
As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to create scenarios that are going to produce emergent behavior of the DUT to discover knowns and unknowns. OpenSCENARIO 2.0 shall enable a demonstration that minimizes residual risk to the DUT. The SOTIF workflow focuses on ensuring the absence of unreasonable risk because of hazards resulting from insufficiencies in the intended functionality or from reasonably foreseeable misuse.
B.8.2 Deriving a simulation scenario from findings of a SOTIF analysis
B.8.2.1 Workflow short description
As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to derive a simulation scenario from findings of a SOTIF analysis. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction.
B.8.2.2 Workflow detailed description
Testing scenarios often do not cover all the safety aspects of a topic/feature. Some safety problems are unfortunately uncovered too late in the development/testing timeline, as they are present even if the system is functioning as defined. Safety issues can be discovered by a SOTIF engineer, a V&V engineer, an AV developer or a scenario creator using OpenSCENARIO 2.0. As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to derive a simulation scenario from findings of a SOTIF analysis. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction.
B.8.2.3 Workflow diagram
B.8.2.4 Steps for discovering scenarios that uncover safety hazards
OpenSCENARIO 2.0 must enable a SOTIF engineer, a V&V engineer, an AV developer or a scenario creator to discover safety issues.
B.8.2.4.1 Handling stub
-
Model the scenario Stub in semiformal language.
A scenario stub is a fragment from an (abstract or concrete) scenario that covers just the moment immediately before a supposed hazard or accidents and contains just the actors and details that are necessary to characterize what supposedly could lead to a hazard. Example: Another car is overtaking the ego vehicle, then merging into the ego vehicle’s lane at short distance and braking hard immediately after. -
Formalize scenario Stub.
-
Generalize scenario Stub.
Generalization refers to the removal of unintended constraining conditions.
Example: "Another car is overtaking the ego vehicle…" → Could it also be a truck or motorcycle that provokes the same outcome?
B.8.2.4.2 Handling context
-
Specify context.
Context is the ODD subset (or operational situation in ISO 26262 [21] language) where the scenario plays.
Example: Driving on a highway at medium speed under reduced visibility conditions. -
Formalize context.
B.8.2.4.3 Handling conditions
-
Specify hazardous condition.
A hazardous condition is a guard condition that indicates that a hazard is present and that shall be monitored later during simulation.
Example: The distance to the predecessor vehicle is lower than 1 m when the ego speed is higher than 10 km/h.
B.8.2.4.4 Creating scenarios
-
Formalize observer condition for simulation.
Semiformal notation can be tables, activity diagrams, template language and so on. Formalizing means to … -
Create new logical scenario if existing scenario pool does not contain appropriate base scenario and environment.
-
Enrich logical scenario.
Enrichment means to add further details that were not present in the scenario stub (for example, weather effects, roadside objects, actors).
B.8.2.4.5 Analyzing and refining
-
Perform sensitivity analysis to find out relevant parameters.
-
Create parameter variants by setting parameters.
-
Add new concrete scenarios with observer conditions to simulation backlog.
At the end of the process the SOTIF engineer, the V&V engineer, the AV developer or the scenario creator can derive a simulation scenario from findings of a SOTIF analysis.
B.8.3 Deriving new hazardous events, system insufficiencies or triggering conditions for SOTIF from findings during a simulation run
B.8.3.1 Workflow short description
As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to derive new hazardous events, system insufficiencies or triggering conditions for SOTIF from findings during a simulation run. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction.
B.8.3.2 Workflow detailed description
Testing scenarios often do not cover all the safety aspects of a topic/feature. Some safety problems are unfortunately uncovered too late in the development/testing timeline, as they are present even if the system is functioning as defined. Safety issues can be discovered by a SOTIF engineer, a V&V engineer, an AV developer or a scenario creator using OpenSCENARIO 2.0. As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to derive new hazardous events, system insufficiencies or triggering conditions for SOTIF from findings during a simulation run. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction.
B.8.3.3 Workflow diagram
B.8.3.4 Steps for discovering scenarios that uncover safety hazards
OpenSCENARIO 2.0 must enable a SOTIF engineer, a V&V engineer, an AV developer or a scenario creator to discover safety issues.
The workflow for a hazardous scenario from simulation is a straight forward workflow. Two of the steps consist of several sub-steps that are described in detail in the next chapters.
B.8.3.4.1 Main steps
-
Log incident.
-
Analyze scenario in parallel.
See Section B.8.3.4.2 for a description of the tasks that can run in parallel. -
Perform sensitivity analysis to find out relevant parameters.
-
Add new logical scenario(s) if existing scenario pool is insufficient to reproduce the incident in all the imagined variants.
-
Extract information.
See Section B.8.3.4.3 for tasks that can run in parallel. -
Export context, triggering conditions (with occurrence frequency estimate) and hazardous behavior to SOTIF analysis tool.
B.8.3.4.2 Analyzing scenario in parallel
The following steps are part of the second step in Section B.8.3.4.1. They can be performed in parallel.
-
Extract underlying base scenario
The logical base scenario contains the context situation (for example, driving on a highway at night), the placement of the objects and actors and the script for the storyline. It has open parameters. The behavior of the ego vehicle (and potentially other actors) is not determined by the scenario, but by its own control functions. -
Extract specific parameters.
-
Handle other actors.
-
Analyze other actors' behavior
In some simulation environments, other actors are given a certain freedom to act, specifying just their driving policy. This induces some non-determinism. To replay the scenario, their behavior must be captured, analyzed and formalized through driving analytics. -
Model other actors' behavior for replay.
-
-
Handle observer.
-
Analyze hazardous behavior of ego vehicle.
-
Formalize observer condition for simulation.
An observer condition is a logical condition that characterizes the presence of a hazard or a faulty behavior of the ego vehicle during simulation.
Example: Vehicle decelerates without a reason; distance to predecessor vehicle is getting too low.
-
B.8.3.4.3 Extracting information in parallel
The following steps are part of the fifth step in Section B.8.3.4.1. They can be performed in parallel.
-
Extract Context Situation for SOTIF analysis.
-
Extract Triggering Conditions for SOTIF analysis.
-
Optional: Estimate occurrence frequency of scenario.
This usually requires external data sources and can be regarded as an activity outside of this use case.
At the end of the process the SOTIF engineer, the V&V engineer, the AV developer or the scenario creator can derive new hazardous events, system insufficiencies or triggering conditions for SOTIF from findings during a simulation run.
B.8.4 Replaying different variants of a critical scenario observed on the road in simulation, to derive new SOTIF insights from it
B.8.4.1 Workflow short description
As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to replay different variants of a critical scenario observed on the road in simulation, to derive new SOTIF insights from it. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction.
B.8.4.2 Workflow detailed description
Testing scenarios often do not cover all the safety aspects of a topic/feature. Some safety problems are unfortunately uncovered too late in the development/testing timeline, as they are present even if the system is functioning as defined. Safety issues can be discovered by a SOTIF engineer, a V&V engineer, an AV developer or a scenario creator using OpenSCENARIO 2.0. As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to replay different variants of a critical scenario observed on the road in simulation, to derive new SOTIF insights from it. This refers to SOTIF and safety hazards that can be present even if the system is functioning as designed, without a malfunction.
B.8.4.3 Workflow diagram
B.8.4.4 Steps for discovering scenarios that uncover safety hazards
OpenSCENARIO 2.0 must enable a SOTIF engineer, a V&V engineer, an AV developer or a scenario creator to discover safety issues.
Processing a hazardous scenario from road testing or field operation is straight forward. See main steps. Two steps contain sub-steps that can be processed in parallel.
B.8.4.4.1 Main steps
-
Ingest incident report, vehicle data logger recordings and camera footage/sensor logs.
-
Extract and formalize.
See Section B.8.4.4.2 for more details. -
Create new logical scenario if existing scenario pool does not contain appropriate base scenario and environment.
The logical base scenario contains the context situation (for example, driving on a highway at night), the placement of the objects and actors, and the script for the storyline. It has open parameters. The behavior of the ego vehicle (and potentially other actors) is not determined by the scenario, but by its own control functions. -
Create scenario output.
See Section B.8.4.4.3 for more details.
B.8.4.4.2 Extracting and formalizing
The following steps are part of the second step in Section B.8.3.4.1. They can be performed in parallel.
-
Formalize Ego Vehicle trajectory and actions.
-
Handle other actors.
-
Identify relevant other actors.
-
Formalize other actors' trajectory and actions.
-
-
Extract relevant road geometry and markings.
-
Extract relevant environmental conditions.
-
Extract relevant objects.
-
Handle observer.
-
Analyze hazardous behavior of Ego vehicle.
-
Formalize observer condition for simulation.
An observer condition is a logical condition that characterizes the presence of a hazard or a faulty behavior of the ego vehicle during simulation.
Example: Vehicle decelerates without a reason; distance to predecessor vehicle is getting too low.
-
B.8.4.4.3 Creating scenario output
The following steps are part of the fourth step in Section B.8.3.4.1. They can be performed in parallel.
-
Prepare scenario.
-
Enrich and set parameters to create concrete scenario.
The concrete scenario is the abstract scenario with selected parameters. In this case it is intended to be a good approximation of what happened on the road. -
Model other actors' behavior for replay.
In some simulation environments, other actors are given a certain freedom to act, specifying just their driving policy. This induces some non-determinism. To replay the scenario, their behavior must be captured, analysed and formalized through driving analytics. -
Optional: Perform sensitivity analysis to find out relevant parameters. Depending on the verification phase it may be enough to reproduce just the concrete scenario that created the issue, or to derive variants to see which parameters are significant, and which other scenarios may also result in hazardous behavior.
-
Optional: Create parameter variants by setting parameters.
-
Optional: Add new concrete scenarios with observer conditions to simulation backlog.
-
-
Export Context, Triggering Conditions (With Occurrence Frequency Estimate) and hazardous behavior to SOTIF analysis tool.
At the end of the process the SOTIF engineer, the V&V engineer, the AV developer or the scenario creator discovers safety hazard thanks to OpenSCENARIO 2.0.
B.8.5 Evaluating of residual risk because of unknown scenario
B.8.5.1 Workflow short description
As a SOTIF safety engineer and/or V&V engineer, AV developer, scenario creator, I can use OpenSCENARIO 2.0 to create scenarios that are going to produce emergent behavior of the DUT to discover knowns and unknowns. OpenSCENARIO 2.0 shall enable a demonstration that minimizes residual risk to the DUT. The SOTIF workflow focuses on ensuring the absence of unreasonable risk because of hazards resulting from insufficiencies in the intended functionality or from reasonably foreseeable misuse.
B.8.5.2 Workflow detailed description
Residual risk of hazardous behavior can be caused by unknown situations.
Demonstrating that this residual risk is acceptably low is a requirement of current AD safety standards (for example, ISO 21448 [20], UL 4600 [23]).
To demonstrate that the risk is low you can use this two-step task:
-
Identify unknown situations leading to hazard.
-
Argue that the risk caused by those situations is still unknown.
You can form the following hypothesis for your argumentation of step 2:
-
Sufficiently dense coverage of the entire ODD by test scenarios reveals almost all such situations.
-
The risk that some situations leading to hazards still slip through is low enough to be acceptable.
OpenSCENARIO 2.0 in combination with OpenODD and OpenXOntology can support this procedure by allowing scenario variant creation to cover the ODD and by providing metrics for sufficient coverage.
The general process to derive abstract scenarios from the ODD and concretize them, and to measure the coverage of the ODD, can be described as follows.
The ODD, as specified in the ASAM OpenODD language, can be understood as a multi-dimensional delimited space. The dimensions are the ODD aspects like road type, lane number, lane with, weather conditions, visibility, presence of infrastructure like guardrails and so on.
The value ranges have one of the following characteristics:
-
They are discrete by nature.
(For example, the number of lanes.) -
They are made discrete by the underlying ontology.
(For example, road type = {highway, country road, city street, off-road}.) -
They are from a continuous range.
(For example, lane width, visibility range.)
With the OpenODD language you can to define equivalence classes (like short visibility = 0..50 m, medium visibility = 50..150 m, and so on).
You can use these inputs to define abstract scenarios that cover all combinations. You can also define scenarios that cover a systematically derived subset of all combinations. The define step can be done automatically. Strategies for automatic definition are known from software testing, see for instance [13], [14], [15].
Note that coverage of the ODD, which is similar to an empty space, is not enough.
On top of the basic ingredients of a scenario that was selected from the ODD categories (for example, road type, weather, number of lanes…) other ingredients are needed:
-
Variable elements
like road decoration (bridges, road signs) -
Static objects
(buildings, obstacles) -
Other traffic participants
and their potential maneuvers -
Associated parameters
(for example, road friction). Catalogs should be available as an input for these parameters.
B.8.5.4 Steps for evaluating of residual risk because of unknown scenario
-
The workflow consists of a few operations that can be done in parallel:
-
Ingest an ODD, potentially in OpenODD compliant format
-
Ingest a catalog of road decoration, static objects, …
-
Ingest a catalog of actors and their potential behaviors
-
-
All of these steps mus be followed by an equivalence class partitioning.
-
A new logical scenario must be added to the existing scenario pool, but only if it does not exist yet.
-
The logical scenario can be enriched with the required objects and actors.
-
All these actions can be done while the definition of the criteria for coverage density is ongoing.
-
Finally, a concrete scenario can be created and added to the simulation backlog.
At the end of the SOTIF review process it is proven that the test case helps to ensure minimum residual risk.