Standardized Labelling and Storage for Scenarios and Vehicle Data
Introduction
On behalf of ASAM we would like to thank all participants of Monday’s ideation workshop, titled "Scenario Storage and Labelling". We had a very high interest, with the key attendance figures summed up below.
Technical Discussions
After presenting the ASAM structure and standardisation process, we set the stage with a high level overview of the scenario workflow. In this workflow we originally identified 3 core topics where we felt standardisation could be of assistance in facilitating a smoother scenario and testing workflow.
- Labelling of objects of interest
- Tagging or labelling of scenario metadata
- Data management & storage of measured scenario data
Guest Speakers
To get the discussion started 9 of the participating companies presented their experiences and the problems they have encountered along the way.
Reclassification of Potential Projects
Using the presentations as a basis the workshop group then redefined and regrouped the core topics into two potential projects:
- Labelling
- Labelling of objects of interest (as identified by sensors)
- Labelling of scenarios (implied as measured or simulated data that is linked to some driving situation)
- Data format for labelled data
- Standardization of raw sensor data
- As of now sensors of the same type (e.g. lidar) all have different output formats, making interchangeability and comparison very difficult
- A high priority demand for standardization of raw sensor data was identified
- As this topic went beyond the original scope of the workshop we will be hosting a further workshop specific to this in early 2020 with a new registration process
Effectively, both original labelling topics are to be pursued, with the addition of a generic labelling data format and a standardization of raw sensor data. It was decided that management and storage of scenario data is more implementation oriented and not yet a priority for standardisation.
The remaining time of the workshop was used to detail out the scope of a project pertaining to 1):
Brainstorming for a Labelling Project
The content will be parallelisable into 3 sub groups as described above. The output of labelling objects of interest will flow into the tags for the scenario labels as a user needs to find scenarios relevant to certain objects. The scenario group will be able to define higher level labels whilst awaiting this input (e.g. environment conditions).
A third sub-group will deal with general data formats for labels, with the goal of abstracting the format away from what is being labelled, thus extending usability.
Below are the captured topics & requirements that arose during the discussions:
- Taxonomy - Describe Objects of interest
- Structure - how will labels be structured (e.g. tree, hierarchy or flat)
- Shall allow for a gradation of granularity of object labels (e.g. Dynamic object → Vehicle → Commercial Vehicle → Truck → [MAKE][Model][other attributes]
- Object relations - are these independent of structure?
- Quality metrics for label structure as well as for labels themselves
- Shall have some approach for defining quality metrics to analyse the quality of data
- Data format
- How would this be structured?
- Data sources
- Shall contain reference to original, raw data sources to allow verification of the data chain
Next Steps
- Deepen AI has volunteered to lead the preparation of a project proposal, supported by the ASAM office
- If you are interested in contributing, please get in touch with us at ASAM (benjamin.engel(at)asam.net)
- Online workshop will be organised for the beginning of quarter 1 2020 to review and refine the project proposal (this will be sent around to all workshop participants as well as anyone that expresses an interest via Mail to the above address)
