# IEEE CEC’2023 Competition on Multiobjective Neural Architecture Search

Background

The advancement of neural architecture search (NAS) has facilitated the automation of deep learning network design, leading to improved performance on various challenging computer vision tasks. Recently, the emerging application scenarios of deep learning, such as autonomous driving, have raised higher demands for network architectures considering multiple design criteria, including the number of parameters/weights, number of floating-point operations and inference latency, among others. From an optimization point of view, the NAS tasks involving multiple design criteria are intrinsically multiobjective optimization problems (MOPs). Hence, it is reasonable to adopt evolutionary multiobjective optimization (EMO) algorithms for tackling them.

Due to the black-box nature of NAS, the complex properties from the optimization point of view (e.g., discrete decision variables, multimodal and noisy fitness landscapes, expensive and many objectives, etc.) pose great challenge to EMO algorithms. Additionally, the progress of adopting EMO algorithms for NAS still lags well behind the progress of the general field of NAS research. Consequently, the main goal of this competition is promoting the research of EMO algorithms on NAS tasks.

In the Multiobjective Neural Architecture Search Competition, participants should use an EMO algorithm designed by themselves to solve the given NAS tasks. The algorithms used can be brand-new or adapted from previously proposed algorithms, but must be relevant to the field of evolutionary computation. We provide an end-to-end pipeline, dubbed EvoXBench, to generate benchmark test suites for EMO algorithms to run efficiently without the requirement of GPUs or Pytorch/Tensorflow. Specifically, EvoXBench covers seven search spaces (NAS-Bench-101, NAS-Bench-201, NATS-Bench, DARTS, ResNet50, Transformer, MNV3), two types of architectures (convolutional neural networks and vision Transformers), two widely-studied datasets (CIFAR-10 and ImageNet), six types of hardware (GPU, mobile phone, FPGA, among others.), and up to six types of optimization objectives (prediction error, FLOPs, latency, energy consumption, among others.). There are two representative benchmark test suites in EvoXBench, i.e., the C-10/MOP and the IN-1K/MOP.

1. C-10/MOP Test Suite

There are nine instances in C-10/MOP tailored for image classification on CIFAR-10 (10-class image classification There are nine instances in C-10/MOP tailored for image classification on CIFAR-10 (10-class image classification dataset). As summarized in Table I, the number of objectives of test instances is from two to eight, including $$f^e$$ (the prediction error of the model), $$f^c$$ (the performance related to model complexity, i.e., weights and FLOPs) and $$f^{\mathcal {H}}$$ (the performance related to hardware devices, i.e., hardware latency, hardware energy consumption and Arithmetic Intensity). The property of C-10/MOP is summarized in Table II.

TABLE I: Definition of the proposed C-10/MOP test suite.

TABLE II: Property of test instances in C-10/MOP.

2. IN-1K/MOP Test Suite

There are also nine instances in IN-1K/MOP tailored for image classification on ImageNet 1K (1000-class image classification dataset). The definition and the property of IN-1K/MOP are summarized in Table III and Table IV. Since the search spaces of ResNet50 and Transformer are resource-intensive and thus unsuitable for efficient hardware deployment, we do not consider hardware-related objectives as architectures from them. Only IN-1K/MOP9 adds a hardware-related objective function, i.e., the latency on a mobile phone.

TABLE III: Definition of the proposed IN-1K/MOP test suite.

TABLE IV: Property of test instances in IN-1K/MOP.

3. Parameter settings

As shown in Table V, the population size is set in correspondence with the number objectives. The algorithm used required 31 independent runs using 10,000 fitness evaluations.

TABLE V: Parameter settings.

Tutorials

1. To better understand EvoXBench and the formulas of NAS problems, you can browse this paper [1] and watch the tutorial video.

2. Install EvoXBench. The installation tutorial refers to the project site of EvoXBench.

3. Run the example code, you can familiarize yourself with how to use EvoXBench. [Python] [Mathlab] [Java]

4. Develop your own EMO algorithm and calculate the indicators with the provided performance indicator tool to check the performance of the algorithm.

For MATLAB users, we recommend using the PlatEMO platform to develop algorithms. The benchmark test suites in EvoXBench have been embedded in PlatEMO, you only need to read the examples to know how to use it. Before running PlatEMO, you need to run “evoxbenchrpc” in the terminal. Then run the MATLAB code of your algorithm.

Evaluation Criteria

In this competition, we use Hypervolume (HV) and Inversion Generation Distance (IGD) as performance metrics. IGD is only used to measure the performance of C-10/MOP1 – C-10/MOP7 since the true Pareto front (PF) is available. For problems derived on the basis of surrogate model (i.e., C-10/MOP8 – C-10/MOP9 and IN-1K/MOP1 – IN-1K/MOP9), the true Pareto Front (PF) are unknown and we cannot calculate IGD. HV is used to measure performance of all test instances. And the settings of reference points for calculating HV metric are provided in Table VI.

All submitted results will be used to calculate HV and IGD in python on the same computer.

TABLE VI: The reference points used on (a) C-10/MOP and (b) IN-1K/MOP test suites.

Submission

Please submit the compressed folder (a .zip file) of your algorithm and send it to zhenyuliang97@gmail.com no later than April 30, 2023.

The .zip file should contain a report, algorithm source code and 18 result files.

The report should contain the following information: (i) title; (ii) names, affiliations, and emails of participants; (iii) description of the algorithm; (iv) mean and standard deviation of HV and IGD values from 31 runs per test instance. Note that only C-10/MOP1 – C-10/MOP7 are considered for the IGD metric, whose true PFs are known from exhaustive evaluations.

The result of each instance should be saved in json format (i.e.,18 .json file). The list of dictionaries is stored in the .json file.

The structure of the .json file is as follows:

[{“run”:1, “X”: [[0, 1, 2, …], [0, 0, 1, …], …, [1, 1, 1, …]], “F”: [[0.19, 39606472], [0.24, 7247656], …, [0.23, 7531312]], “IGD”: 0.01, “HV”: 0.9},
{“run”: 2, “X”: [[0, 1, 2, …], [0, 0, 1, …], …, [1, 1, 1, …]], “F”: [[0.19, 39606472], [0.24, 7247656], …, [0.23, 7531312]], “IGD”: 0.01, “HV”: 0.9},
…,
{“run”: 31, “X”: [[0, 1, 2, …], [0, 0, 1, …], …, [1, 1, 1, …]], “F”: [[0.19, 39606472], [0.24, 7247656], …, [0.23, 7531312]]}, “IGD”: 0.01, “HV”: 0.9}]


where “run” is the sequence number of each run test instance; “X” denotes an encoded network architecture (i.e., decision vector); “F” is the objective function on validation mode; “IGD” and “HV” are performance metric. It is not necessary to provide metric values in the .json file, but we recommend that the submitted files include this information.

Awards

IEEE CEC 2023 conference certificates and prize money will be awarded to the winners of this competition(1st to 3rd place).

Questions

If you have any question to the competition , please contact Zhenyu Liang（zhenyuliang97@gmail.com）or join the QQ group to ask questions (Group number: 297969717).

Organizers

Name: Zhenyu Liang

Affiliation: Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. (Email: zhenyuliang97@gmail.com)

Short Bio: Zhenyu Liang received the B.Sc. from Southern University of Science and Technology in 2020. He is currently a master student in the Department of Computer Science and Engineering, Southern University of Science and Technology. His main research interests are evolutionary algorithms and evolutionary multi- and many-objective optimization.

Name: Zhichao Lu

Affiliation: School of Software Engineering, Sun Yat-sen University, China. (Email: luzhichaocn@gmail.com)

Short Bio: Zhichao Lu is currently an Assistant Professor in the School of Software Engineering at Sun Yat-sen University, China. He received his Ph.D. degree in Electrical and Computer Engineering from Michigan State University, USA, in 2020.

His current research focuses on the intersections of evolutionary computation, learning, and optimization, notably on developing efficient, reliable, and automated machine learning algorithms and systems. He received the GECCO-2019 best paper award.

Name: Ran Cheng

Affiliation: Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China. (Email: ranchengcn@gmail.com)

Short Bio: Ran Cheng is currently an Associate Professor with the Southern University of Science and Technology (SUSTech), China. He received the PhD degree in computer science
from the University of Surrey, UK, in 2016.

Dr Ran Cheng serves as an Associated Editor/Editorial Board Member for several journals, including: IEEE Transactions on Evolutionary Computation, IEEE Transactions on Cognitive and Developmental Systems, IEEE Transactions on Artificial Intelligence, IEEE Transactions on Emerging Topics in Computational Intelligence, etc. He is the recipient of the IEEE Transactions on Evolutionary Computation Outstanding Paper Awards (2018, 2021), the IEEE CIS Outstanding PhD Dissertation Award (2019), the IEEE Computational Intelligence Magazine Outstanding Paper Award (2020). Dr Ran Cheng is the Founding Chair of IEEE Computational Intelligence Society (CIS) Shenzhen Chapter. He is a Senior Member of IEEE.

Name: Yaochu Jin

Affiliation: Faculty of Technology, Bielefeld University, Germany. (Email: yaochu.jin@uni-bielefeld.de)

Short Bio: Yaochu Jin is an Alexander von Humboldt Professor for Artificial Intelligence endowed by the German Federal Ministry of Education and Research, Chair of Nature Inspired Computing and Engineering, Faculty of Technology, Bielefeld University, Germany. He is also a Distinguished Chair, Professor in Computational Intelligence, Department of Computer Science, University of Surrey, Guildford, U.K. He was a “Finland Distinguished Professor” of University of Jyväskylä, Finland, “Changjiang Distinguished Visiting Professor”, Northeastern University, China, and “Distinguished Visiting Scholar”, University of Technology Sydney, Australia. His main research interests include evolutionary optimization, evolutionary learning, trustworthy machine learning, and evolutionary developmental systems.

Prof Jin is presently the President-Elect of the IEEE Computational Intelligence Society, and the Editor-in-Chief of Complex & Intelligent Systems. He was named by the Web of Science as “a Highly Cited Researcher” from 2019 to 2022 consecutively. He is a Member of Academia Europaea and Fellow of IEEE.

References

[1] Z. Lu, R. Cheng, Y. Jin, K. C. Tan and K. Deb, “Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment,” in IEEE Transactions on Evolutionary Computation, doi: 10.1109/TEVC.2022.3233364.