FLBench
Background
Federated learning is a new machine learning paradigm. The goal is to build a machine learning model from the data sets distributed on multiple devices, so-called an isolated data island, while keeping their data secure and private. Most existing federated learning benchmarks work manually splits commonly-used public datasets into partitions to simulate real-world isolated data island scenarios. Still, this simulation fails to capture real-world isolated data island’s intrinsic characteristics. This paper presents a federated learning (FL) benchmark suite named FLBench. FLBench contains three domains: medical, financial, and AIoT. By configuring various domains, FLBench is qualified to evaluate federated learning systems and algorithms’ essential aspects, like communication, scenario transformation, privacy-preserving, data distribution heterogeneity, and cooperation strategy. Hence, it becomes a promising platform for developing novel federated learning algorithms. Currently, FLBench is open-sourced and in fast-evolution. We package it as an automated deployment tool.
FLBench Framework
FLBench framework including four parts.
Input Data: Most of the current researches on FL are carried out on the simulation scenario, which is constructed by commonly used dataset such as CIFAR-10. However, there is a vast difference between the commonly used dataset data and the real-world scenario data in data type and data mode. This considerable difference leads to that FL algorithms developed based on simulated scenarios cannot be migrated to real-world scenarios. We collect data from the three most concerning scenarios to solve this issue, including medicine, finance, and AIoT. Besides, a particular data pre-process tool is necessary for medicine data since medicine data need special processing.
Scenario Configuration: To achieve the robustness and multi-faceted evaluation of the FL algorithms, we propose a scenario configuration function. First, we analyze the current innovation of FL researches and then classify the innovation directions of FL into the following categories: communication, scenario transformation, privacy-preserving, data distribution heterogeneity, cooperation strategy. Second, for each innovation direction on each domain, we provide a basic configuration according to the natural distribution of data and an API to modify the configuration to simulate various scenarios according to requirements.
Scenario: Benchmark has two functions: first, it provides an open and fair comparison; second, it will provide a research basis for later researchers to develop more advanced algorithms and determine the selection of some important parameters. Thus, we construct two kinds of scenarios: fixed scenarios and customized scenarios. We modify the basic configuration mentioned above to achieve customized scenarios.
Automated Deployment Tool: We will update FLBench step by step to make it adapt to future development needs. Besides, we continue to expand the benchmark and provide more scenarios and related APIs. We hope that more people will join our benchmark research, which will make our benchmarks suite more perfect and comprehensive.
FLBench Implement
Currently, FLBench contains: four datasets (medicine: ADNI ,MIMIC-III; finance: Adult dataset ; AIoT: iNaturalist-User-120k), one basic configuration file (Alzheimer’s diagnosis scenario configuration). The Alzheimer’s diagnosis scenario configuration can provide various scenarios for NO-IID (data distribution heterogeneity) research in the medicine domain.
FLBench is a fully open and evolving benchmark; next, we will provide 3∗3 =9 datasets for three domains (medicine, finance, and AIoT), and 3 ∗ 3 ∗ 5 = 45 basic configuration files on different research aspects, including communication, scenario transformation, privacy-preserving, data distribution heterogeneity, and cooperation strategy. Each configuration file can provide various scenarios according to the requirements of the specific research.
Publication: FLBench: A Benchmark Suite for Federated Learning
Liang Y, Guo Y, Gong Y, Luo C, Zhan J, and Huang Y. FLBench: A Benchmark Suite for Federated Learning[C]//Intelligent Computing and Block Chain: First BenchCouncil International Federated Conferences, FICC 2020, Qingdao, China, October 30–November 3, 2020, Revised Selected Papers. Springer Nature, 166.
Abstract: Federated learning is a new machine learning paradigm. The goal is to build a machine learning model from the data sets distributed on multiple devices--so-called an isolated data island--while keeping their data secure and private. Most existing federated learning benchmarks work manually splits commonly-used public datasets into partitions to simulate real-world isolated data island scenarios. Still, this simulation fails to capture real-world isolated data island's intrinsic characteristics. This paper presents a federated learning (FL) benchmark suite named FLBench. FLBench contains three domains: medical, financial, and AIoT. By configuring various domains, FLBench is qualified to evaluate federated learning systems and algorithms' essential aspects, like communication, scenario transformation, privacy-preserving, data distribution heterogeneity, and cooperation strategy. Hence, it becomes a promising platform for developing novel federated learning algorithms. Currently, FLBench is open-sourced and in fast-evolution. We package it as an automated deployment tool. The benchmark suite is available from https://www.benchcouncil.org/flbench.html.