BROWSE
Detail
Conference Paper
Why Is Human-Preference Optimization Difficult for Co-Speech Gesture Generation? An Audited Study of Rewards, Generator Support, and Evaluation
Cited - time in 
Share 
Share
- Authors
- David J. Cha, SungYoon Lim, Dohyung Kim, Jaehong Kim, Youngwoo Yoon
- Issue Date
- 2026-09
- Citation
- European Conference on Computer Vision (ECCV) 2026 Workshop : Interactive Social Avatars (ISA), pp.1-14
- Publisher
- Interactive Social Avatars (ISA)
- Language
- English
- Type
- Conference Paper
- Abstract
- One utterance can admit many plausible gestures, but supervised imitation typically regresses toward the single recorded motion-capture sequence. Preference learning could instead reward any generated motion that people judge natural and well aligned with the speech. We examine whether preference signals can improve strong pretrained gesture generators, including methods that update generator weights and a final setting that updates no generator weights. The final system freezes GestureLSM and steers only its initial noise using a runtime-reference-free naturalness-and-alignment reward. The reward improves development-set naturalness accuracy from 71.9% to 89.6%, yet two larger human studies detect no reliable preference advantage over the untrained generator. A training-free best-of-32 control raises the learned score (higher is better by construction) without a reliable human gain. Controlled interventions further show that the frozen generator mainly exposes local variations of a confined family of coarse motion patterns. We attribute the result to two coupled limits: the reward loses fine-ranking validity on optimized outputs, and the desired behavior may be absent from the generator’s reachable outputs. We derive corresponding validation requirements for gesture rewards, fixed generators, and human evaluation.
- Keyword
- co-speech gesture generation, human-preference optimization, reward models, flow matching, human evaluation
- KSP Keywords
- Development-set, Final setting, Flow matching, Gesture generation, Human Evaluation, Motion capture, Motion patterns, Preference optimization, Reference-free, Two coupled, capture sequence