BROWSE
Detail
Journal Article
Evaluating reinforcement learning from human feedback for task-oriented dialogue systems
- Authors
- Hyeok-Min Gwon, Yohan Lee, Jin-Xia Huang, Jonghyuk Lee
- Citation
- ETRI Journal, Early Access
- ISSN
- 1225-6463
- Publisher
- 한국전자통신연구원
- Language
- English
- Type
- Journal Article
- Abstract
- Reinforcement learning from human feedback (RLHF) has shown strong potential for aligning language models, but its role in task-oriented dialogue (TOD) remains unclear. In TOD, models are typically trained with local turn-level supervision, while system behavior is evaluated through broader interaction-level properties. This mismatch becomes more challenging in online settings, where explicit dialogue-level rewards and human preference annotations are unavailable. In this work, we study whether RLHF can be usefully applied to TOD under this limitation. We consider two task-annotation regimes, partially annotated and fully annotated TOD data, and construct pseudo-preference pairs using empirical ranking heuristics motivated by prior work on synthetic feedback and model-based ranking signals. We then train reward models on the constructed pairs and optimize dialogue policies with Preference policy optimization (PPO) using simulator-generated online trajectories. Experiments on MultiWOZ 2.1 show that the proposed RLHF approach consistently improves corpus-based evaluation over supervised baselines, while simulator-based effects remain mixed and backbone-dependent.
- Keyword
- deep learning, machine learning, natural language processing, reinforcement learning from human feedback, task-oriented dialogue
This work is distributed under the term of Korea Open Government License (KOGL)
(Type 4: : Type 1 + Commercial Use Prohibition+Change Prohibition)
(Type 4: : Type 1 + Commercial Use Prohibition+Change Prohibition)
