BROWSE
Detail
Conference Paper
Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning
Cited 0 time in 
Share 
Share
- Authors
- Daegun Yoon, Sangyoon Oh
- Issue Date
- 2024-05
- Citation
- International Symposium on Cluster, Cloud and Internet Computing (CCGrid) 2024, pp.307-316
- Language
- English
- Type
- Conference Paper
- Abstract
- Communication overhead is a major obstacle to scaling distributed training systems. Gradient sparsification is a potential optimization approach to reduce the communication volume without significant loss of model fidelity. However, existing gradient sparsification methods have low scalability owing to inefficient design of their algorithms, which raises the communication overhead significantly. In particular, gradient build-up and inadequate sparsity control methods degrade the sparsification performance considerably. Moreover, communication traffic increases drastically owing to workload imbalance of gradient selection between workers.To address these challenges, we propose a novel gradient sparsification scheme called ExDyna. In ExDyna, the gradient tensor of the model comprises fined-grained blocks, and contiguous blocks are grouped into non-overlapping partitions. Each worker selects gradients in its exclusively allocated partition so that gradient build-up never occurs. To balance the workload of gradient selection between workers, ExDyna adjusts the topology of partitions by comparing the workloads of adjacent partitions. In addition, ExDyna supports online threshold scaling, which estimates the accurate threshold of gradient selection on-the-fly. Accordingly, ExDyna can satisfy the user-required sparsity level during a training period regardless of models and datasets. Therefore, ExDyna can enhance the scalability of distributed training systems by preserving near-optimal gradient sparsification cost. In experiments, ExDyna outperformed state-of-the-art sparsifiers in terms of training speed and sparsification performance while achieving high accuracy.
- KSP Keywords
- Build up, Communication overhead, Communication volume, Distributed training, High accuracy, Model fidelity, Non-overlapping, Training speed, Training system, Workload imbalance, accurate threshold