Home

Edge AIBench is a scenario-based benchmark suite as a part of scenario benchmarks in AIBench. Edge AIBench uses the scenario-based method and summarizes four typical edge AI scenarios: autonomous vehicle, ICU patient monitor, Smart Home, and Autonomous Vehicle. Edge AIBench provides nine edge AI workloads and give reference implementations of them.

The rapid growth of the number of the client side devices brings great challenges to computing power, data storage, and network, considering the variety and quantity of these devices. The traditional way to solve these problems is the cloud computing framework. Moreover, with the development of 5G network technology, the Internet of Things(IoT) has been another solution. Moreover, edge computing shows a trend of rapid development these years, which combines cloud computing and IoT framework. In the edge computing scenarios, the distribution of data and collaboration of workloads on different layers are serious concerns for performance, security, and privacy issues. So for in edge computing benchmarking, we must take an end-to-end view, considering all three layers: client-side devices, edge computing layer, and cloud servers.

Figure 1 Edge AIBench covers the whole layers

After a general investigation of a set of edge computing applications, we find those applications have a great diversity in many aspects. Thus, we need to consider several operational characteristics these scenarios have.
Moreover, edge cloud computing has recently received great attention from the industry. For this reason, we also consider industrial tasks and requirements. AI tasks are usually involved in edge computing scenarios, such as image classification, face em-bedding, speech recognition, etc.

Figure 2 Edge AIBench covers the whole layers

Therefore, AI applications need to be considered as the represented workloads in our benchmark suite.
Based on the above concerns, we think AI scenari-os are the primary scenarios of edge computing. Ad-ditionally, we summarize six properties of edge AI computing in this section.
To organize our benchmark suite, we adopt a scenario-based benchmark view. The scenario-based idea means extracting primary scenarios which can represent the characteristics. Also, those scenarios need to cover all applications' complexity in edge computing. Edge AIBench is a scenario benchmark considering scenarios' characteristics and typical ap-plications.
In Edge AIBench, we extract four typical scenarios which can represent those characteristics: autonomous vehicle, ICU patient monitor, surveillance camera, and smart home. We think they can represent the complexity of edge computing. In addition, Edge AIBench provides an end-to-end application benchmarking framework, including train, validation and inference stages. Table 1 shows the component benchmarks of Edge AIBench. Edge AIBench provides an end-to-end application benchmarking, consisting of train, inference, data collection and other parts using a general three-layer edge computing framework.

Table 1. The summary of Edge AIBench

Program Name Edge AI Scenarios Models Datasets Implementation
Lane Detection Autonomous Vehicle LaneNet Tusimple, CULane Pytorch/Caffe
Traffic Sign Detection Autonomous Vehicle Capsule Network German Traffic Sign Recognition Keras
Death Prediction ICU Patient Monitor LSTM MIMIC-III Tensorflow/Keras
Decompensation Prediction ICU Patient Monitor LSTM MIMIC-III Tensorflow/Keras
Phenotype Classification ICU Patient Monitor LSTM MIMIC-III Tensorflow/Keras
Person Re-identification Surveillance Camera DG-Net Market-1501 PyTorch
Action Detection Surveillance Camera ResNet18 UCF101 PyTorch/Caffe
Face Recognition Smart Home FaceNet/SphereNet LFW/CASIA-Webface Tensorflow/Caffe
Speech Recognition Smart Home DeepSpeech2 LibriSpeech Tensorflow

ICU Patient Monitor. ICU is the treatment place for critical patients. Therefore immediacy is significant for ICU patient monitor scenario to notify doctors of the patients’ status as soon as possible. The dataset we use is MIMIC-III. MIMIC-III provides many kinds of patients data such as vital signs, fluid balance and so on. Moreover, we choose heart failure prediction and endpoint prediction as the AI benchmarks.
Surveillance Camera. There are many surveillance cameras all over the world nowadays, and these cameras will produce a large quantity of video data at all times. If we transmit all of the data to cloud servers, the network transmission bandwidth will be very high. Therefore, this scenario focus on edge data preprocesses and data compression.
Smart Home. Smart home includes a lot of smart home devices such as automatic controller, alarm system, audio equipment and so on. Thus, the uniqueness of the smart home includes different kinds of edge devices and heterogeneous data. We will choose two AI applications as the component benchmarks: speech recognition and face recognition. These two components have heterogeneous data and different collecting devices. These two component benchmarks both collect data on the client side devices (e.g., camera and smartphone), infer on the edge computing layer and train on the cloud server.
Autonomous Vehicle. The uniqueness of the autonomous vehicle scenario is that the high demand for validity. That is to say, it takes absolute correct action even without human intervention. This feature represents the demand of some edge computing AI scenarios. The automatic control system will analyze the current road conditions and make a corresponding reaction at once. We choose the road sign recognition as the component benchmark.

A Federated Learning Framework Testbed. We have developed an edge computing AI testbed to provide support for researchers and common users, which is publicly available from http://www.benchcouncil.org/testbed.html. Security and privacy issues become significant focuses in the age of big data, as well as edge computing. Federated learning is a distributed collaborative machine learning technology whose main target is to preserve the privacy. Our testbed system will combine the federated learning framework.

Figure 3 Edge Computing Testbed with Federated Learning