ANALYSIS: EVENT-BASED VISION Event-based vision is well suited to eye-tracking applications, for example in human-machine interfaces
only transmit information when there is a change in the scene’s brightness. This real-time response is especially beneficial in rapidly changing lighting conditions, such as sudden transitions from light to darkness or vice versa.
Adapting event sensors for IoT Event-based vision sensors acquire sparse data, making them suited for edge vision applications with limited on-board compute resources. However, event sensors’ unconventional data format, non-constant data rates and non-standard interfaces have restrained wider adoption. In response to that, the latest
generation event sensors, such as Prophesee’s recently introduced GenX320, are designed with the explicit goal to improve integrability and usability in embedded at-the-edge vision systems, including AI accelerators and Edge SoCs (system-on-chips). As a result, new sensors include adaptations such as event data pre-processing and formatting, data interface compatibility and low-latency connectivity to different processing platforms including low- power neuromorphic processors. The GenX320, for example, features multiple data pre-processing, filtering and formatting functions, variable MIPI and CPI interfaces and a hierarchy of power modes, facilitating operability in power- sensitive vision applications. Despite their operating efficiency, event
sensors must be further optimised for low-power operation at levels practical for IoT systems. Offering a hierarchy of power modes and application-specific modes of operation can help achieve better energy efficiency, especially in ‘always on’ applications. On-chip intelligent power management modes and on-chip power management can further improve sensor flexibility and usability. In Prophesee’s case, this reduces power consumption to as low as 36µW and smart wake-on-events
www.imveurope.com | @imveurope
are enabled. Deep sleep and standby modes can also be supported. Other specific considerations for an
event sensor targeting IoT include: • Low latency microsecond resolution timestamping of events with flexible data formatting • Easy integrability/interfacing with standard SoCs with multiple integrated event data preprocessing, filtering, and formatting functions to minimise external processing overhead • MIPI or CPI data output interfaces offering low-latency connectivity to embedded processing platforms, including low-power microcontrollers and modern neuromorphic processor architectures • Sensor-level privacy-enabled thanks to the event sensor’s inherent sparse frameless event data with inherent static scene removal.
A wider range of use cases By focusing on the integrability with IoT platforms, as well as market-specific requirements in terms of power and size, event-based sensors become appealing for more applications in AR/VR, edge AI, automotive, smart consumer electronics, safety, and mobile.
Specific use cases include: foveated
rendering for more immersive and natural experiences in AR/VR headsets and gaming; eye tracking for human machine interfaces; safety applications such as driver monitoring systems (DMS); always- on capabilities for security and safety uses (e.g. fall-detection cameras); and gesture/ hand tracking for immersive and intuitive interfaces and displays. Other applications in the AR/VR space include inside-out tracking and constellation tracking based on flickering LEDs, which allows high- precision object or controller tracking. Early adopters of Prophesee’s latest
offering include Zinn Labs, which is developing a next generation of gaze tracking systems for AR/VR devices. In
this use case, low latency is critical and an event sensor such as Prophesee’s GenX320 meets the demands of eye and gaze movements that change on millisecond timescales. Unlike traditional video-based gaze tracking pipelines, such a sensor can track features of the eye with a fraction of the power and compute required for full-blown computer vision algorithms, bringing the footprint of the gaze tracking system below 20mW. The small package size of the new sensor also allows an event-based vision sensor to be applied to space-constrained head- mounted applications in AR/VR products. Other product developers see significant
value in the privacy benefit of event sensors. Xperi, for example, has a “privacy by design” philosophy for its driver monitoring system (DMS). Event-based vision takes that to an even more secure level by allowing scene understanding without the need to have explicit visual representation of the scene. By capturing only changes in every pixel, rather than the entire scene as with traditional frame-based imaging sensors, Xperi’s algorithms can derive knowledge to sense what is in the scene, without a detailed representation of it. Xperi has developed a proof-of-concept demo showing that DMS is fully possible using neuromorphic sensors. It has shown that by using a 1MP neuromorphic sensor, its systems can deliver similar performance to an active NIR illumination 2MP vision sensor-based solution. Applying event-based vision to enable hands-free interaction via highly accurate gesture recognition and hand tracking capabilities is appealing to Ultraleap and its TouchFree application. The sensor’s ability to operate in challenging environmental conditions, at very efficient power levels and with low system latency, enhances the overall user experience and intuitiveness of its touch-free UIs. With the advantages of robustness, low power consumption, latency and high dynamic range, Ultraleap sees that event sensing can be extended to more types of applications and devices, including battery- operated and small form factor systems, proliferating hands-free use cases. Event-based vision is well on its way to establishing itself as a paradigm that will create a new standard in the market. Over the past several years, its implementations have evolved to meet a wider range of market needs (as evidenced by Prophesee, one of the early commercial pioneers, having just released its fifth-generation sensor). And by continuing to adapt and address the requirements of many applications, we will see more event-based cameras all around us. I
DECEMBER 2023/JANUARY 2024 IMAGING AND MACHINE VISION EUROPE 17
Prophesee
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32
