상세정보
학술대회
Towards Understanding Architectural Effects on Knowledge Distillation
- 발행일
- 202010
- 출처
- International Conference on Information and Communication Technology Convergence (ICTC) 2020, pp.1144-1146
- 협약과제
- 20HS3300, 부하분산과 능동적 적시 대응을 위한 빅데이터 엣지 분석 기술 개발, 이용주
- 초록
- Knowledge distillation is a promising model compression solution, which adopts an apprenticeship (teacher-student) learning approach. Even referring to the same teacher network, it has reported different distillation performance according to a used student network architecture in current works. To tackle this issue, we investigate how scaling depth and width over layers influences learning capability based on distillation. Our experiment results show that depth scaling is a more determinant factor than width for selecting a smaller network in knowledge distillation.
- KSP 제안 키워드
- Experiment results, Learning Capability, Learning approach, Model compression, Network Architecture, knowledge distillation