BROWSE
Detail
Journal Article
Cross-corruption distillation for robust instruction following under text corruptions
- Authors
- Yongjin Bae, Kyoungman Bae
- Issue Date
- 2026-09
- Citation
- ETRI Journal, v.48, no.5, pp.852-867
- ISSN
- 1225-6463
- Publisher
- John Wiley & Sons
- Language
- English
- Type
- Journal Article
- Abstract
- Traditional knowledge distillation transfers information from a large teacher model to a compact student by aligning static output distributions or internal representations for fixed inputs. Such formulations overlook how predictions change in response to input perturbations, even though robustness is critical for deployment. We interpret distillation as the transfer of functional response behavior under input variations and propose cross-corruption distillation (CCD), which aligns corruption-induced differences in teacher and student output via a Kullback–Leibler objective. Rather than matching absolute logits, CCD distills how the teacher redistributes its prediction mass under perturbations, transferring teacher-conditioned input sensitivity without architectural correspondence. On diverse multimodal benchmarks, CCD performs competitively, often outperforming conventional logit-based distillation. It also slows performance degradation as perturbation strength increases and remains stable across corruption strategies and training schedules. These results position CCD as a response-aware distillation framework integrating robustness directly into the distillation objective.
- Keyword
- behavior-level distillation, knowledge distillation, robustness, trustworthy AI, vision-language models
- KSP Keywords
- Functional response, Knowledge Distillation, Language Models, Strength increases, Teacher Model, Traditional knowledge, performance degradation, response behavior
This work is distributed under the term of Korea Open Government License (KOGL)
(Type 4: : Type 1 + Commercial Use Prohibition+Change Prohibition)
(Type 4: : Type 1 + Commercial Use Prohibition+Change Prohibition)
