Supplement: Semiconductors Generative AI moves to the edge
Latest systems-on-chip with high AI performance and efficiency enable locally processed large language models. By Amit Badlani, director of generative AI and robotics, Ambarella
A
I is constantly in the news as one advancement follows another, delivering capabilities that just a decade ago were only to be found in
books and science fiction movies. However, so far the performance requirements have limited the most recent generative AI (GenAI) advancements to applications with cloud-based processing.
Until recently, the primary focus of GenAI has been based around power hungry servers that were tasked with developing and training large language models (LLMs). While this was, in itself, a breakthrough, it represented just the first step on a much more significant journey. Currently, there is significant focus on pushing widespread technology development at the edge as this will allow GenAI to be deployed in a wide range of applications that currently enjoy limited benefit from this technology. Specifically, these use cases will significantly increase the number of embedded applications for GenAI in markets such as smart cities, industrial automation, robotics and autonomous driving.
For GenAI to be successful at the edge the ‘three Ps’ of AI must be delivered – privacy, performance and power efficiency. Increasingly, developers are demanding a fourth ‘P’ – productivity – allowing them to bring their ideas to market quickly. Delivering the necessary integration at the edge brings technical challenges around energy efficiency, on-device fine- tuning and reliability to the forefront. Difficult enough on their own, these hurdles must be overcome in a solution that can be delivered to developers at an acceptable cost.
To meet these often conflicting challenges, the preferred solution is emerging as tailored systems-on-chip (SoCs).
Energy efficient SoCs for GenAI Recognizing the need for highly capable yet energy efficient AI processors in edge applications, Ambarella offers a range of AI
40 May 2024
SoCs for a multitude of GenAI applications at the edge.
In most high performance multi-modal analytics use cases for edge applications, the first step is to capture high quality video under all lighting conditions. To that end, an on-chip image signal processor (ISP) is needed that can provide excellent image quality, even in low-light conditions. Likewise, high-contrast scenes must be captured with great clarity and detail using a processor that is also capable of advanced high dynamic range (HDR), providing a very versatile solution for security video analytics, autonomous mobile robots (AMRs) and autonomous driving. Additionally, with efficient encoding, that same SoC can stream high- resolution video at very low bit rates. In addition to supporting LLM processing, Ambarella’s proprietary CVflow AI engine that is integrated into
Components in Electronics
its SoCs can run several neural networks (NNs) in parallel for accelerating classical computer vision algorithms with minimal power consumption. Simplifying the porting of pre-developed neural networks, Ambarella’s Cooper developer platform contains a comprehensive suite of tools. Ambarella’s GenAI SoC solutions are scalable ranging from the N1 family, supporting up to 34 billion parameter multi-modal LLMs, to the CV7 family supporting smaller vision language models (VLM) that are up to three billion parameters.
Ambarella’s flagship N1 series of SoCs is a natural choice for applications demanding greater processing power at the edge. It can process multiple high resolution camera streams and run LLMs with up to 34 billion parameters for multi-modal analytics, or connect a greater number of peripheral devices. Building
upon Ambarella’s expertise in achieving maximum AI performance per watt, the N1 SoCs offer advanced neural network computation using the integrated neural vector processor (NVP), an advanced image signal processor, a dense stereo and optical flow engine, 16 Arm Cortex-A78AE CPUs, and a GPU for user visualizations only—all in a single SoC. Despite this performance, N1 SoCs require a fraction of the power-per-inference of industry- leading GPU solutions.
The N1 is well suited for delivering GenAI to multiple applications, including industrial robotics, smart cities, intelligent healthcare imaging and diagnostics, multi-camera AI processing hubs, edge AI servers running multi-modal LLMs, and autonomous fleet telematics. On-device multi-modal LLM processing facilitates GenAI functionality, such as smart contextual searches of security footage;
www.cieonline.co.uk
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64
