BROWSE
Detail
Journal Article
AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model
Cited 7 time in 
Share 
Share
- Authors
- Jeong Hun Yeo, Minsu Kim, Jeongsoo Choi, Dae Hoe Kim, Yong Man Ro
- Issue Date
- 2024-04
- Citation
- IEEE Transactions on Multimedia, v.26, pp.6462-6474
- ISSN
- 1520-9210
- Publisher
- Institute of Electrical and Electronics Engineers
- Language
- English
- Type
- Journal Article
- Abstract
- Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements. VSR is regarded as a challenging task because of the insufficient information on lip movements. In this article, we propose an Audio Knowledge empowered Visual Speech Recognition framework (AKVSR) to complement the insufficient speech information of visual modality by using audio modality. Different from the previous methods, the proposed AKVSR 1) utilizes rich audio knowledge encoded by a large-scale pretrained audio model, 2) saves the linguistic information of audio knowledge in compact audio memory by discarding the non-linguistic information from the audio through quantization, and 3) includes Audio Bridging Module which can find the best-matched audio features from the compact audio memory, which makes our training possible without audio inputs, once after the compact audio memory is composed. We validate the effectiveness of the proposed method through extensive experiments, and achieve new state-of-the-art performances on the widely-used LRS3 dataset.
- KSP Keywords
- Audio Features, Linguistic information, Speech information, Visual Speech Recognition, large-scale, state-of-The-Art