Mitigating Catastrophic Forgetting via Sparse Replay in Deep Networks

Abstract

Deep neural networks trained sequentially on multiple tasks suffer from catastrophic forgetting—the abrupt loss of previously acquired knowledge upon learning new information. Experience replay, which stores and revisits past training examples, is among the most effective mitigation strategies, yet conventional replay buffers impose substantial memory overhead that limits scalability. This paper presents a comprehensive survey and empirical analysis of sparse replay buffer methods that achieve competitive or superior anti-forgetting performance while maintaining memory budgets orders of magnitude smaller than full experience replay. We formalize the continual learning problem and taxonomy of approaches, then focus on sparse replay strategies including coreset selection, gradient-based sample prioritization, compressed exemplar storage, generative replay with distillation, and hybrid regularization-replay methods. Through systematic experiments on Split CIFAR-100, Split ImageNet, Permuted MNIST, and Sequential Omniglot benchmarks, we demonstrate that sparse replay with as few as 1–5 exemplars per class achieves 85–95% of full replay performance. We analyze the interplay between buffer size, selection strategy, and task similarity, providing practical guidelines for deploying continual learning systems under memory constraints.