BROWSE
Detail
Conference Paper
Beyond Semantics: Exploring Context Ranking under Knowledge Conflicts
Cited 0 time in 
Share 
Share
- Authors
- Sanghyeon Cho, Myeongjin Lee, Jeong Heo, Jong-hun Shin, Kiyoung Lee, Soojong Lim
- Issue Date
- 2026-06
- Citation
- Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD) 2026, pp.191-203
- Publisher
- Springer
- Language
- English
- Type
- Conference Paper
- Abstract
- We study knowledge conflicts in the Generate-then-Read pipeline, where large language models generate semantically similar but factually inconsistent contexts that can confuse the reader model. To reproduce and analyze such scenarios, we construct ConflictsQA, a dataset designed to simulate knowledge-conflict situations. It extends open-domain question–answer pairs with contexts generated by both GPT-4o and open LLMs such as LLaMa and Gemma, where GPT-4o provides ground-truth True contexts and open models produce both True and plausibly incorrect False contexts. Using ConflictsQA, we conduct a comprehensive evaluation of both embedding-based and cross-encoder models to assess their ability to distinguish factually reliable contexts under conflicting knowledge. Our evaluation includes lightweight models such as GTE and BGE, as well as LLM-based models like Qwen3 in both embedding and cross-encoder models. This setup enables us to compare how different model architectures and sizes handle the balance between semantic similarity and factual consistency. We also explore model pruning under fixed memory constraints. Pruned models show competitive results when evaluated on contexts generated by GPT-4o, where semantic cues are relatively distinct, but exhibit large performance drops on fully conflicting data. This indicates that pruning preserves semantic discrimination while weakening factual reasoning.
- Keyword
- Knowledge Conflicts, Text Ranking, Generate-then-read
- KSP Keywords
- Comprehensive Evaluation, Language Models, Lightweight model, Semantic similarity, conflicting data, ground truth