Mnemonics Training

This repository contains the PyTorch implementation for "Mnemonics Training: Multi-Class Incremental Learning without Forgetting". If you have any questions on this repository or the related paper, feel free to create an issue or send me an email.

Summary

• Introduction
• Dependencies
• Running Experiments
• Citation
• Acknowledgements

Introduction

Multi-Class Incremental Learning (MCIL) aims to learn new concepts by incrementally updating a model trained on previous concepts. However, there is an inherent trade-off to effectively learning new concepts without catastrophic forgetting of previous ones. To alleviate this issue, it has been proposed to keep around a few examples of the previous concepts but the effectiveness of this approach heavily depends on the representativeness of these examples. This paper proposes a novel and automatic framework we call mnemonics, where we parameterize exemplars and make them optimizable in an end-to-end manner. We train the framework through bilevel optimizations, i.e., model-level and exemplar-level. We conduct extensive experiments on three MCIL benchmarks, CIFAR-100, ImageNet-Subset and ImageNet, and show that using mnemonics exemplars can surpass the state-of-the-art by a large margin. Interestingly and quite intriguingly, the mnemonics exemplars tend to be on the boundaries between classes.

Figure: The t-SNE results of three exemplar methods in two phases. The original data of 5 colored classes occur in the early phase. In each colored class, deep-color points are exemplars, and light-color ones show the original data as reference of the real data distribution. Gray crosses represent other participating classes, and each cross for one class. We have two main observations. (1) Our approach results in much clearer separation in the data, than random (where exemplars are randomly sampled in the early phase) and herding (where exemplars are nearest neighbors of the mean sample in the early phase). (2) Our learned exemplars mostly locate on the boundaries between classes.

Dependencies

• Python >= 3.6
• PyTorch >= 0.4.0
• Pillow >= 6.2.1
• tensorboardX
• tqdm
• scipy

Running Experiments

Clone this repository

cd ~
git clone [email protected]:yaoyao-liu/mnemonics-training.git

Processing the datasets

Download the ImageNet (ILSVRC2012) source file.

Process ImageNet-Sub and ImageNet:

cd ~/mnemonics-training/main/process_imagenet
python generate_imagenet_subset.py
python generate_imagenet.py

CIFAR-100 will be downloaded automatically.

Download models

Download the models for CIFAR-100, ImageNet-Sub and ImageNet:

cd ~/mnemonics-training/main
sh ./script/download_ckpt.sh

You may also download the checkpoints on Google Drive.

Evaluation on our models

Run evaluation code with our models：

cd ~/mnemonics-training/main
sh run_eval.sh

We're checking the training code. It will be updated later.

Citation

Please cite our paper if it is helpful to your work:

@inproceedings{liu2020mnemonics,
author    = {Liu, Yaoyao and Su, Yuting and Liu, An{-}An and Schiele, Bernt and Sun, Qianru},
title     = {Mnemonics Training: Multi-Class Incremental Learning without Forgetting},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
pages     = {12245--12254},
year      = {2020}
}

Acknowledgements

Our implementation uses the source code from the following repositories:

• Learning a Unified Classifier Incrementally via Rebalancing

• iCaRL: Incremental Classifier and Representation Learning

• Dataset Distillation

• Generative Teaching Networks