The Seven attributes of an Open Data Format for LiDAR Software

Frequently Asked Questions

• What is TLV encoding and why do LiDAR data formats use it?

  TLV (Type-Length-Value) encoding is a serialization scheme where each data element is tagged with a type identifier, a length field, and the value itself. Because every element is self-describing, a decoder can skip any element it does not recognize without corrupting the rest of the message. This makes TLV a natural fit for LiDAR pipelines, where software versions across a fleet rarely update in lockstep: a newer sensor stream reaching an older decoder simply skips the unfamiliar fields and keeps processing without error. Outsight's SHIFT platform relies on this kind of open, versioning-tolerant data format to maintain compatibility across a multi-vendor LiDAR hardware ecosystem spanning Hesai, RoboSense, Ouster, Velodyne, and Seyond sensors, all feeding the same real-time processing pipeline.

• What is OSEF and how does it relate to LiDAR processing output?

  OSEF (Open Serialisation Format) is Outsight's open data format for streaming processed 3D scenes. At each LiDAR frame (typically 20 frames per second), the OSEF structure is a tree containing a timestamp and all associated attributes: tracked object arrays, classification metadata, and optional streams such as Ego-Motion. The format is a core output layer of the SHIFT platform, which processes LiDAR point clouds through a sub-50ms end-to-end pipeline before serialising results into OSEF for downstream consumption. Because the format is open and published without licensing restrictions, any downstream application or integration partner can implement a conformant reader without negotiating proprietary access.

• Why do proprietary sensor output formats create problems in multi-sensor deployments?

  Each LiDAR manufacturer ships its own binary protocol. A deployment using sensors from three vendors requires three separate decoding drivers, and any firmware update from one vendor can silently break the corresponding driver in the processing stack. In practice this slows integration work, multiplies maintenance surface, and makes cross-sensor fusion harder. A single normalized output format from the processing layer removes that complexity by presenting a uniform data stream regardless of which physical sensors are upstream. This is why Outsight built the SHIFT platform to be LiDAR-native with multi-vendor compatibility across hardware from Hesai, RoboSense, Ouster, Velodyne, Seyond, Innoviz, and many more, abstracting vendor-specific protocols behind a common data layer so that mixed-sensor deployments can be maintained and scaled without driver fragmentation.

• What is Ego-Motion data in a LiDAR stream, and when should it be disabled?

  Ego-Motion data describes the movement of the sensor itself, covering its position, velocity, and orientation change between frames. It is critical for mobile platforms such as autonomous vehicles or robots, where the sensor is constantly moving and its motion must be subtracted to isolate object movement. In fixed-infrastructure deployments (sensors mounted on ceilings, poles, or gantries), the sensor does not move, so streaming Ego-Motion on every frame wastes bandwidth and decoder cycles. An adaptable data format lets operators suppress those fields for infrastructure use cases. Outsight's infrastructure-based Physical AI approach, where LiDAR sensors are fixed in the environment rather than mounted on moving entities, is a practical example of why toggling Ego-Motion off is a meaningful optimization for real-world deployments at scale.

• How does backward compatibility in a data format affect large-scale LiDAR rollouts?

  In a large installation, dozens to hundreds of edge processing units rarely receive software updates simultaneously. A backward-compatible format ensures that a unit running an older software version can still parse frames produced by a newer one, and vice versa, because unknown fields are skipped rather than causing parse failures. Without backward compatibility, a staged rollout becomes a coordination problem: every node must be updated before any node can use new format features, which is operationally impractical at airport or city scale. This is precisely the challenge Outsight addresses in deployments like Dallas Fort Worth, where a large number of infrastructure-mounted LiDAR sensors must continue operating across incremental software updates without interrupting the real-time data pipeline that feeds the Motional Digital Twin.

• Can a single LiDAR data stream carry output from dozens of fused sensors simultaneously?

  A format designed for flexibility can encapsulate the combined output of many sensors in one stream without changing the format structure itself. In large-area deployments such as airport terminals or city intersections, tens of LiDAR sensors are fused into a single shared point cloud, and the resulting tracked-entity stream is delivered as one normalized feed. Outsight's SHIFT platform applies exactly this principle: across deployments like Dallas Fort Worth and Paris-Charles de Gaulle, dozens of infrastructure-mounted LiDAR units are fused in real time, yet the receiving application sees one stream of classified, positioned entities regardless of how many physical sensors contributed to that picture.

• What makes a data format open, according to the definition used for LiDAR processing software?

  An open data format is one that has a freely available published specification which places no restrictions, monetary or otherwise, upon its use. Because the specification is public, others can implement conformant software that reads and writes data in that format. For LiDAR, this means any developer can build tools that consume the processing software's output. Open formats are presented as imperative for 3D LiDAR to reach mainstream use in scalable applications.

• Why did early LiDAR manufacturers use proprietary data formats in the first place?

  Proprietary data formats and protocols made sense when LiDAR first appeared, because they shortened the time to market of new hardware products delivering raw data. The first LiDAR prototypes were not intended for production purposes, and the burden of decoding each specific format was on the early users. For the technology to reach mainstream use in scalable use cases, the article argues that 3D LiDAR data must be accessible through open data formats.

• Why should a LiDAR data format allow data elements to appear in any order inside a message?

  Allowing data elements to be placed in any order inside the message body is the article's example of robustness in a LiDAR data format. According to Outsight, it greatly simplifies decoding drivers and limits bugs. It also facilitates backwards compatibility, where new features and data fields can be added while older or unknown elements are safely skipped. This matters because software versions are released frequently and several versions will likely cohabit in the same network and system.

• What is the trade-off between simplicity and performance when designing a LiDAR output format?

  A LiDAR processing format should be simple to use, and parsing data on the system must be straightforward and efficient. The article notes that this is easier to say than to do, because more complex formats can be more efficient in some specific use cases. Simplicity is therefore a trade-off between usability, time to market and performance. It is the second of the seven attributes the article lists for an open LiDAR data format, after openness and before robustness, backwards compatibility, efficiency, adaptability and flexibility.

• How does an efficient LiDAR data format protect the real-time performance of point-cloud processing?

  A LiDAR point-cloud software should avoid wastage in processing time, which slows down processes and can even compromise real-time capabilities. An efficient data format therefore keeps overhead processing time to a minimum and optimises bandwidth. In Outsight's Shift Perception, each processed LiDAR frame, typically at 20 frames per second, is delivered in the OSEF format. Efficiency is one of the seven attributes the article sets for an open LiDAR data format.

• What does it mean for a LiDAR data stream to be adaptable to real-time user requests?

  An adaptable format lets the system react to real-time user requests without changing its structure and decoding patterns, so only relevant data is sent at all times. In application-agnostic software like Outsight's, each user needs only part of the available outputs. An adaptable solution streams just the information the use case requires. This optimises both bandwidth and decoding time for every application. For example, in an infrastructure project with LiDARs in a fixed position, streaming Ego-Motion information on each frame is not useful.

• How is each processed LiDAR frame structured in the Shift Perception OSEF output?

  At each processed LiDAR frame, typically at 20 frames per second, the OSEF output of Shift Perception is structured as a tree. The tree contains the timestamp information and all its attributes, for example Ego-Motion and Tracked Objects' information. OSEF, the Open SErialisation Format, uses TLV encoding, a scheme used for optional informational elements. Outsight states that OSEF is open, simple, robust, backwards-compatible, efficient, adaptable and flexible.

• Why does hardware-agnostic LiDAR processing software need a single output data format?

  A key advantage of LiDAR processing software that is agnostic to the sensing hardware is its ability to output a unique data format regardless of the input device. Many projects require a combination of sensors from different manufacturers. Processing software such as Outsight's Shift platform converts raw LiDAR data into actionable insights that applications can use, for example to precisely detect and track people. The article argues that open data formats are imperative for 3D LiDAR to reach mainstream, scalable use.

• How can I check whether a LiDAR processing software's output format is truly open and standard?

  The article proposes seven characteristics to check. The format should be open, usable without proprietary intellectual property, and simple to parse. It should be robust, with elements in any order, and backwards-compatible, so new fields can be added while old or unknown ones are safely skipped. It should also be efficient in processing and bandwidth, adaptable to stream only relevant data, and flexible enough to handle any volume of data without changing the format.

• How does an open LiDAR data format reduce legal and platform lock-in risk for users?

  An open system functions without proprietary intellectual property, which lowers the risk of using it. According to the article, open formats can be used without restrictions and support movement from one technical platform or environment without compromising legal agreements. The intellectual property is also typically publicly available, which provides greater security and accessibility to a bigger range of software and technical platforms. Openness is the first of the seven attributes the article describes.

• Can people flow data be integrated into my existing BI or operations systems?

  Yes, a modern flow monitoring platform should expose every metric and event through a documented API so it can feed BI tools, airport operational databases, building management or security systems. Outsight's Spatial Intelligence Platform does this, and can also ingest business data such as flight schedules or point-of-sale data to enrich its analytics. For example, terminal flow KPIs can be analysed by airline or flight, and POS data can be combined with dwell times to model conversion. Dashboards, alerts and reports are also available for teams that prefer ready-to-use tools.