OpenLABEL Concept Paper

OpenLABEL is a standard for annotation formats, which can be used for data streams and scenarios created during the development of automated driving features. For an automated vehicle to operate within its design domain a understanding of its surroundings is crucial. The vehicle senses its environment by using different sensors (e.g. cameras, lidar, radar). The data from the sensors must be interpreted to create a processable image of the world, based on this image the vehicle can choose its action. For the development of such features a ground truth is necessary. This ground truth is created by annotating the raw data. With in the data the following can be annotated:

When labeling an object it is possible to use different methods. On the one hand the labelling method depends on the data, but also on the use case. Labeling a vehicle in Lidar data requires a 3 dimensional method, while labeling the same car in a video image requires only a 2 dimensional method.

During the development of an automated vehicle it is very important to identify the scenarios that fit the use case and the design domain the vehicle is supposed to operate in. For that purpose it is useful to label scenarios according to their content and relevance to the project.

Developing an automated vehicle is a very complex tasks and usually many parties are involved in such a project. This makes an exchangeable format very valuable.

In the OpenLABEL concept project it is the goal of the project group to create concepts to establish a common format that can provide the industry with a standard for the annotation of sensor data and scenarios. The format will be machine processable and still be human readable. Also the group creates a basis to build a user guide. The user guide will explain to any future user how to use the available labeling methods, that are supported by OpenLABEL. The concept paper created in this project will serve as a basis for a follow up standard development project.

This concept paper serves as a source for a future development of the OpenLABEL standard. In the OpenLABEL Concept Project the basic concepts for a future annotation standard are created. To this the project has four different working groups. Each working group created their own concept and aligned them within the project and with the other OpenX Activities at ASAM.

The Concept project has four workpackages:

The OpenLABEL Concept has a close link to the OpenXOntology. This link will ensure that the OpenLABEL Taxonomy requirements are met in the upcoming OpenX Core Domain Model.

Relation to other Standards:

This section details the annotation format of OpenLABEL. The format is key to make OpenLABEL flexible enough to host different labeling use cases, ranging from simple object-level annotation in images (e.g. with bounding boxes), to complex multi-stream labeling of scene semantics (e.g. with actions, relations, contexts). The annotation format is then understood as the materialization of labels in files or messages, that can be stored or exchanged between machines. The format shall address a number of requirements:

The annotation format management shall also define which properties are mandatory and which optional, types of variables, and serialization mechanisms to create persistent content, such as files or messages.

Next sections detail all these aspects, with examples and definitions.

One modern approach to define data structure is to create a JSON schema, which is itself a JSON document that contains descriptions and constraints on the structure and content of JSON files, and it also provides a data model.

A JSON schema can be used to validate the content of a JSON file, guaranteeing that the file follows the constraints and structure dictated by the schema. Also, schemas can be used by programming languages to create object-oriented structures which facilitate manipulation, edition and access to information of a JSON file.

The annotation format of OpenLABEL is then proposed to be hosted on a detailed JSON schema file, and as a consequence, annotation files will be JSON files following the schema. Appended to this document the draft openlabel_schema_json-v1.0.0.json can be found.

In OpenLABEL, a scene can be either a subset of the reality that needs to be described for further analysis, or a virtual situation that needs to be materialized. In the former case, reality is typically perceived by sensors, which get discrete measures of magnitudes from the scene at a certain frequency. In the latter, sensors can be ignored, and the scene described by its components and logical sequence.

Several concepts conform the basis of the OpenLABEL format. As it will be shown, these pieces constitute the foundations to create rich descriptions of scenes, either as an entire block (e.g. serialized as a file), or frame-by-frame (e.g. serialized as message strings).

The basic serialization of an OpenLABEL JSON string (prettified), with just administrative information and versioning is:

Next subsections show how to add different concepts (Elements, Frames and Streams) and show examples for relevant use cases considered in this concept-paper. For the sake of space and readability, partially collapsed JSON strings will be shown.

This section provides details of the on the object_data primitives defined in OpenLABEL annotation format. Most of them are self-explanatory, as they represent primitives types like string, num (single number, floating point precision), vec (array of numbers), bool (boolean).

Geometric types are more complex. Next sub-sections describe their format.

This section provides detail on the synchronization of multiple streams and their time information frames.

Labels can be produced to be related to specific streams (e.g. cameras, LIDAR). When multiple such streams are present and labels need to be produced for several of them (e.g. bounding boxes for images of the camera, and cuboids for the point clouds of the LIDAR), then, a synchronization and matching strategy is needed.

Determining the synchronization of the data streams (e.g. images and point clouds) correspond to the data source set-up, and not to the annotation stage. For example, the data container may contain precise HW timestamps for images and point clouds, and in addition, the correspondence between frame indexes for multiples cameras (e.g. Frame 45 of camera 1 corresponds, because of proximity in time, to Frame 23 of camera 2, maybe because they have different frequency or have started with some delay).

Therefore, when producing labels for such different, the annotation format need to allocate space and structure for such timing information, such that all labels are perfectly and easily associated to their corresponding data and time.

The JSON schema defines the frame data containers, which correspond to "Master Frame Indexes".

An OpenLABEL format data stream may contain sensor data from multiple sensors and multiple styles of label information associated with that sensor data. It needs to be clear how data from different sensors is related, how the label data relates to the sensor data and how the sensor data relates to the real world. Multiple coordinate systems with transforms between them are used to achieve this.

For example, a data stream may contain images from two forward facing cameras arranged as a stereo pair together with LIDAR return data from a single top mounted mechanically rotating LIDAR, all on a moving vehicle also containing a GNSS/INS system.

In this example it is necessary to understand the positions and orientations of the 4 sensors (two cameras, one LIDAR, one GNSS/INS) with respect to each other and with respect to the vehicle.

The data stream may also, for example, contain label data for 2D object bounding boxes for the left camera plus 3D object bounding boxes which are a human /algorithm annotator’s best estimate of where in the world objects are based on both the stereo camera information and the LIDAR information. It is necessary to understand how the 2D and 3D labels relate to each other.

It is also necessary in many cases to understand how the sensor data and the labels relate to the real world as represented by the absolute positions and orientations the GNSS/INS system can provide, or as represented by map data. For example, it is often necessary to understand where objects are in relation to the road structure as provided by a map.

More specifically, for a camera it is desirable to understand the position of the ray of light that generated the value of each pixel in the image. This requires knowledge of the position and orientation of the camera in the world together with the positions of objects/road/etc in the world. Any distortion introduced by the camera lens also needs to be understood. For a LIDAR system, again, it is desirable to understand the point in the world which each LIDAR return is generated from. This requires knowledge of the position of the LIDAR device in the world, and the position of objects hit by the LIDAR in the world. Also since in this example the LIDAR is a relatively slow scanning mechanical device it is necessary to understand the motion of the LIDAR device through the world in order to be able to understand the positions of all the LIDAR returns relative to each other (and the world).

In general it is convenient to have a coordinate system for each sensor, together with a coordinate system for the vehicle and also a coordinate system for the world.

Sensor data is naturally stored in the coordinate system for the sensor that produced it. Labels that are specific to a sensor, e.g. 2D bounding boxes for a camera, are also naturally stored in the coordinate system of the relevant sensor. Labels that are related to world coordinates, like the fused camera and LIDAR 3D bounding box in this example are most naturally stored in the world coordinate system (that the human did the annotation in).

It may be thought that providing a fixed small number of coordinate frames, e.g. sensor(s), vehicle, world would be sufficient, however this is not the case. Different sensor set ups and different system level choices can lead to many more coordinate systems being needed with significant variation between systems.

For example, with a stereo camera system, it is usual to have a pre-processing stage that takes the ‘raw’ camera images and undistorts and rectifies them to produce a new pair of ‘rectified’ images. This removes the lens distortion and aligns the horizontal lines of both cameras (so that the rectified images fit an ideal pinhole camera model and searching for the same point in the scene in both cameras can be performed by just searching along a line). These rectified images appear to come from virtual cameras with slightly different orientations and focal lengths than the physical cameras. This difference can be represented by having an ‘image sensor’ coordinate system and a ‘(virtual) camera’ coordinate system and a transform between them. Some systems will record the ‘raw’ images and the transforms necessary to generate the rectified images (and may or may not also record the rectified images). Some systems will record only the rectified images.

For example, with a LIDAR system, it is usual to have a pre-processing stage that takes one rotation of raw LIDAR return data and converts it into a point cloud. This removes the rotating over time nature of the LIDAR scan by taking into account how the LIDAR sensor has moved (i.e. how the vehicle has moved) over the time taken for the LIDAR to make one rotation. This ‘untwisting’ process generates a point cloud in a coordinate system that is fixed in the world. In some systems the point cloud is in a coordinate system that is locked to where the LIDAR was at the start (or end) of the scan; in other systems it is in a coordinate system that is fixed in the world but can drift over time relative to a map (an ‘odom’ type coordinate system); and in some systems it is in a coordinate system that is absolutely fixed in the world (a GNSS or map type coordinate system).

Another reason different systems may use different coordinate systems is that the maps they use may be different. For example, some systems may use GNSS type coordinates (an elliptical coordinate system); some systems may use UTM coordinates (rectangular coordinates that are appropriate for local areas of the earth); and some systems may use country specific mapping systems (such as the UK Ordinance Survey map coordinates). Other systems may use proprietary maps such as those created by dense LIDAR surveys of an area together with a LIDAR localisation method.

Yet another reason that different systems may use different coordinate systems is that they may have made different system level choices for how to represent motion of the vehicle through the world. For example, many systems use the model that the ROS (Robot Operating System) middleware uses where the chain of transforms includes the ‘odom’ coordinate system which is a coordinate system that is approximately fixed in the world but can drift over time. In this model the transform sequence is typically: sensor → vehicle → odom → map → earth. In this case the system generates its own idea of motion (odometry) through the world using local sensor data like camera or inertial sensors, then this ‘odom’ coordinate system is localised by placing it on a map using some global sensor data like GNSS. The benefit of this system is that it can gracefully handle situations where there is a loss of global sensor data such as when a vehicle enters a tunnel and loses GNSS signal. In this case the vehicle will be able to detect its movement through the tunnel, but only approximately and over time the vehicle’s idea of its position on a map will drift from its actual position on a map, then when the vehicle emerges from the tunnel it will regain GNSS signal and be able to correct its idea of position on the map. This is represented by the vehicle motion being smooth in the odom coordinate system, but there being discontinuous changes of the transformation between the odom and map coordinate systems.

Yet another system level choice for different systems using different coordinate systems might be choices like whether to compensate for tilt of the road. Some systems may have an additional coordinate system (at some point in the transform sequence) that is referenced to ‘down’ as detected by using accelerometers (with compensation for acceleration due to motion).

For all these reasons the OpenLABEL standard provides a method to describe an arbitrary number of coordinate systems and a method to describe the transforms between those coordinate systems.

However, despite the ability to describe an arbitrary set of coordinate systems, there are some coordinate systems that are commonly used in many systems and so are defined by the standard. The coordinate systems with fixed definitions include: “vehicle-iso8855”, “odom”, “map-UTM”, “geographic-wgs84”. Whenever these names are used for a coordinate system, they shall have the meaning defined in the standard.

It is also important to note that the transformations between coordinate systems can vary over time. In the example above the odom to map transform varies as the vehicle emerges from the tunnel. Even transforms that might appear fixed, because they are rigidly connected like the transformation from camera to vehicle, can in fact vary over time. For example, a camera system may have a dynamic re-calibration system which may change the transformation between the camera sensor coordinate system and the (virtual) camera coordinate system, thus changing the transform between the camera and the vehicle.

The OpenLABEL standard therefore provides a method to describe transforms which are fixed for all time, which vary occasionally (at a specific frame), and which vary continuously (every frame).

Concepts

The key concepts are:

Each coordinate-system has a textual name and a uid that is used to reference it within the OpenLABEL JSON data stream. A small number of names are reserved and refer to pre-defined coordinate systems specified in the standard. All other names are user defined. Each coordinate-system is defined by either being associated with a sensor, or by being the source or target of a transform, where there is a sequence of such transforms that ends in either a sensor or a pre-defined coordinate system.

Each transform is one of a fixed number of types. The types supported are:

Pre-defined coordinate systems

The following coordinate system names are predefined, and wherever used have the following meaning:

Typical transform trees

There are several sets of coordinate systems (blue boxes) and transforms (blue lines) between them that are commonly used.

For example, a ROS based system with the sensors described in the example system in the introduction might have the following transform tree:

A set of data captured from a dash-cam (single camera plus GPS) might look like:

A single camera with no other data, with with the movement of the camera deduced by structure from motion, might look like:

When labelling objects with in data streams different geometries are necessary. depending on the typ of sensor stream either 2D or 3D geometries are needed. Therefore the OpenLABEL Standard will provide a set of primitives that can used to label objects and areas in the sensor streams.

The format described in chapter 1 supports the following geometry types :

There are several ways to describe spatial rotations, each of which has its own advantages and disadvantages. This chapter discusses these methods as well as their differences.

Description:

Bounding Boxes are used to label objects and entities detected by sensors mounted e.g on a vehicle. There are different types of bounding boxes 2D/3D.
Ususally the primary sensor for 2D bounding boxes is the camera.
Ususally the primary sensor for 3D bounding boxes is the lidar or radar.

Using bounding boxes is a cost and time efficient way to label data. It is easiery to draw boxes over detailed "painting" of areas in the data. Data sets that are labeled with bounding boxes are also cheaper to process in terms of processing power and use less space in storage.

Depending on the target Machine Learning Network datasets labeled with bounding boxes are mandatory.

General Description: <what is semantic segementation in general?>

Sematic image segmentation, also called pixel-level classification, is the task of clustering parts of an image together which belong to the same object class. Technically, it means assigning to each pixel a value/code corresponding to a certain class of interest (object/entity category).

For convenient human visual consumption, this is typically achieved through the designation of a color code for each class. The information of e certain pixel belonging to a certain category is thus expressed by assigning to that pixel a specific RGB value, which visually represents that category.

It is important to notice how the semantic segmentation task treates objects as stuff, which is amorphous and uncountable. Multiple objects of the same class are treated as a single entity. Thus, no information exists about specific instances of a class. Cars are all assigned a blue color code (for example) and are treated as being part of the same amorphous "car stuff".

Semantic segmentation annotations follow the form of the objects and have no fixed shape. Manually, this is usually achieved by drawing refined polygons around the regions of interest, or by painting the region of interest through a paintbrush-like feature. The result is a precise mask that isolates only the object of interest and no surrounding pixels.

In the 2D Annotation Space this method provides the highest accuracy of the objects. However, this comes at increased costs in comparison to other annotation methods. Furthermore, segmentations take up more time during the labeling process than other 2D annotation methods and thus have lower throughput.

Current methods of scenario definition focus on the representation of scenarios for the purpose of scenario execution (either in simulation or real-world) or for training Machine Learning systems.

When working with scenarios it is often found that additional data beyond that which can be represented in a scenario definition is necessary to facilitate their use, discoverability, management, and portability. Similarly, there some types of data, where it is not always easy or efficient to attempt to extract it from a scenario, and for some types of data it is impossible due to the limitations of the scenario definition language being used.

Two types of scenario generation methods exist: bottom-up approach and top-down approach.

Top-down scenario generation methods using predefined languages:

Bottom-up scenario generation methods:

To illustrate how the goals for the scenario labels can look like. the below image demonstrates the workflow. In the left bucket all available labels are gathered, this includes, higher level labels, scenario content related labels, odd labels etc. As soon as a scenario needs to be labeled the applicable labels are selected from the "bucket". Depending on the purpose/objective of the scenario the selected labels might differ. Now the scenario (with all its necessary files) is stored along the annotation data (labels).

Now one use for the labels is shown in the below image is to form search queries to get the corresponding scenarios. In the below example the scenario label searched is "school zone", all matching scenarios are returned. This example only illustrates a very simple use case, when using the labels also relations between labels might be part of the search query, e.g. right turn at a playground. How the relation between the two labels [right-turn] and [play-ground] are realized needs to be determined in the follow-up project.

Multiple definitions and meanings of the term “scenario” exist today within the context of Automated Driving. It is not the intention of the current document to shed light on the best definition, nor it is to increase the body of knowledge around scenarios. Our aim is to provide a high-level definition of the term “Scenario” which is relevant to the current concept document for OpenLABEL standard.

The concept of scenario we are interested in “involves telling a “story” about a more or less detailed sequence of events, including at least one situation within a scene, its scenery and dynamic elements, and ongoing activities of one or more actors.” [Geyer et. al., 2014]

This “story” unfolds in a specific domain of discourse which is relevant to us: the one of road traffic. It is then useful to decompose this domain into its main elements in order to provide a conceptual mental model that will prove to be a useful tool of orientation throughout this and the other OpenX standards.

For Scenarios we need the following 3 conceptual constructs + 1 additional introduced later for clarity:

Conceptually, a scenario is nothing more than a description of a time interval detailing a set of events and activities performed by a relevant set of dynamic agents. Such a description is grounded in its environment and scenery, and may include the goals of one or more of the participants.

Operationally, scenarios can be described and substantiated at many levels of abstraction and semantics, either quantitatively through rigorous mathematical definitions and state-space variables, qualitatively, through the use of natural language, or a mixture of both, through the use of human and machine-readable languages like SDLs (Scenario Description Languages) that usually have both a procedural and declarative nature.

So, while “scenarios” usually refer to the same conceptual construct and domain of discourse (that of sequences of events happening on a certain road environment), they can take shape in many different “artifacts” such as specific file formats (e.g - openScenario 1.x, etc.), Domain Specific Languages files (OSC 2.0, M-SDL, etc.), written natural languages sentences, or even raw sensor recordings enriched with additional metadata.

In sum, It is important to disambiguate and clarify the distinction between “scenario” as a concept and its domain of discourse, which is unique, and the embodiment of such domain into a specific artifact, which can take many forms, for different purposes.

Usually, as of today, in the Automotive domain, such an embodiment process - which we call “Scenario Generation” - happens along two main axes: top-down and bottom-up.

It is useful to distinguish two streams in Scenario Artifacts Generation: top-down and bottom-up.

Regardless of the generation method, it becomes clear that many different objects and artifacts exist that deal in some way with the concept of a scenario and the 3 constructs of the domain of discourse introduced previously.

The common artifacts that embody a description of scenario in a human and machine readable manner are often referred to as Scenario Description Languages (SDLs), and these are frequently used in the context of simulating scenarios. The ASAM OpenSCENARIO SDL is particularly relevant as it sits within the same family of OpenX standards as OpenLABEL. OpenSCENARIO 1.x is a relatively concrete SDL, whereas OpenSCENARIO 2.x is broader, and can describe both concrete scenarios and more abstract scenarios.

OpenLABEL attempts to be independent of the specific SDL used to represent scenarios; this means that it should provide a consistent set of labels to be queried, regardless of the details of the representation capabilities of the SDL being used. Given this independence from SDLs, the labels in OpenLABEL may capture a certain level of detail about the content of the scenario - related to the 3 constructs - (for example, characteristics of the location where it occurs, the vehicles involved, and perhaps the maneuvers they execute).

It then becomes useful to introduce another construct in our conceptual model, “Additional Labels”, which does not deal with the purely ontological concepts of the scenario domain of discourse - its specific content or narrative -, but rather introduces the aforementioned additional concepts and information.

The above table summarizes our thought and provides a birdseye view over the conceptual mental model of the domain of discourse we introduced, and its interaction with multiple scenario artifacts. OpenSC1.x files cover concepts related to the constructs of “Dynamic Agents” and possibly “Environment” of our mental model in a pretty concrete manner, and may point to OpenDRIVE files which concretely and describe elements of “Scenery”. OSC2.x instead, while sharing some similarities to Osc1.x, provides a broader spectrum of abstraction and covers more use cases. In fact it can be considered a superset of OSC1.x.

Scenario data can be either recorded real-world data, or an abstracted representation of a scenario intended for use in scenario recreations. In either case it is assumed that the scenario data is stored in file(s) in a common format, and in addition to the core scenario data these files may themselves contain labels about the scenario. Therefore the source files for the scenario must contain the scenario data (which describes the observable features and evolution of the scenario), but (depending on the format) may also contain labels.

For abstracted scenario representations, the common file formats are often referred to as Scenario Description Languages (SDLs), and these are frequently used in the context of simulating scenarios. OpenLABEL attempts to be independent of the specific SDL used to represent scenarios; this means that OpenLABEL should provide a consistent set of data to be queried, regardless of the details of the representation capabilities of the SDL being used.

Given this independence from SDLs, the scenario labels in OpenLABEL should capture a certain level of detail about the scenario (for example, characteristics of the location where it occurs, the vehicles involved, and perhaps the maneuvers they execute). In some ways it can therefore resemble a highly-abstract SDL itself. This is both unavoidable and desirable — if OpenLABEL held no information about the scenario data, it would not be of any use for finding scenarios according to core observable characteristics.

In order to gain further clarification of the role of OpenLABEL and also its relations to other ASAM projects such as OpenSCENARIO, OpenXOntology, an example of the basic structure of a book is used to illustrate.

Within a book, there are content section, preface, individual chapters. In this case OpenLABEL would be the book content section, it provides clear link to each individual chapters. The preface can be considered as a high level scenario description language which describe the chapters in a very compact format. Each individual chapters can be considered as a detail low level scenario description language. The quality or the logic of the actual written content can be related to the ontology.

The complexity of the information associated with scenarios can raise challenge to the scenario labelling, at a high level the information requires labelling can be divided into derived data and non-derived data. Derived data represents domain specific information that can be derived directly from the scenarios, whereas non-derivable data corresponding to externally added information that cannot be derived directly from the scenarios. Examples of the derived data include road layout, vehicle manoeuvre, traffic signs, etc. Examples of non-derived data include authorship, ADAS feature under test, licenses, etc.

While the elements within the non-derived data normally does not contain domain specific relationships or class hierarchies, the elements within the derived data resemble the domain model and contain complex relationships and hierarchies, therefore it needs to be divided further into sub classes. In this meta-model for scenario labelling, the non-derived data falls within the Admin Labels category.

The Operational Design Domain (ODD) for Automated Driving Systems (ADSs) contains all the required attributes that describe the ODD of ADSs. ODD refers to the operating conditions under which an ADS can perform safely, it covers the non-behavior attributes of an scenario and it is highly desirable to enable ODD based scenario labelling and searching. Defining ODD is usually the first step of the development process of an ADS, it allows clear definition of the capabilities and limitations of the ADS, and it enables clear communication to the end user. ODD is also crucial for testing and validating an ADS, test engineers would create test scenarios that are tailored towards the ODD boundaries. Even during runtime (could be either in simulation of real-world), being able to monitor whether the ADS is operating within its ODD is of key interests for various stakeholders such as insurance provider, ADS developer and end users. Therefore the second category of the meta-model for scenario labelling will be ODD Labels.

Within a scenario, the ODD elements cover the non-behavior content. Behavior related content such as vehicle manoeuver or event triggers are not inside the operating design domain, therefore the third category of the meta-model for scenario labelling needs to cover Behavior Labels.

In order for OpenLABEL to be able to accommodate the requirements of various users, the meta-model shall include custom labels, these are the labels that are unique to individuals and should not be part of the standard. The organization of these custom labels, ie. whether custom labels will be embedded into the three categories under specific hierarchical position or whether they should be placed all under a separate custom label category, will be left for the standardization project.

Labelling a scenario is potentially labour-intensive, so OpenLABEL should not mandate that every label has a value for every scenario. However, many use cases will rely on certain labels being present: therefore, the concept of LabelSets should be used to allow users to specify which labels are mandatory for them. The LabelSet mechanism needs to be developed by the standardization project (see the open issue Optional labels and purpose bound LabelSets)

TODO: introduce a mechanism to reference labels from external standards to differentiate between OpenLABEL labels and external labels. TODO: add more examples to the topic to fully understand how the meta model shall be used. TODO: The user guide needs to clearly explain how to label scenarios, so they can be found using search engines

OpenLABEL should include definitions for the values of each label. For all labels that map to OpenXOntology concepts, these definitions should align closely with the OpenXOntology definitions. A toy example could be:

However, in many cases it will be hard to find a single definition for a value that meets the needs of all users. In this Concept paper, high-level descriptions are given for each suggested label, but the final OpenLABEL standard should additionally provide definitions for values.

Complexities with providing unambiguous definitions include:

There is the need to identify and, where needed, define and standardize a set of labels that can robustly serve this use case. We identified three main axis that will serve the purpose of organizing the storage, search and retrieval of scenarios artifacts within a database:

While the ODD labels and Behavior labels are derived, the contextual/admin metadata is mainly non-derived, the details of this will depend on the format of the underlying scenario data.

Proposal: