ETRI-Knowledge Sharing Plaform

KOREAN
논문 검색
Type SCI
Year ~ Keyword

Detail

Conference Paper Automatic Evaluation of English-to-Korean and Korean-to-English Neural Machine Translation Systems by Linguistic Test Points
Cited 1 time in scopus Share share facebook twitter linkedin kakaostory
Authors
Sung-Kwon Choi, Gyu-Hyeun Choi, Youngkil Kim
Issue Date
2018-12
Citation
Pacific Asia Conference on Language, Information and Computation (PACLIC) 2018, pp.1-8
Language
English
Type
Conference Paper
Abstract
BLEU is the most well-known automaticevaluation technology in assessing theperformance of machine translation systems.However, BLEU does not know which parts ofthe NMT translation results are good or bad.This paper describes the automatic evaluationapproach of NMT systems by linguistic testpoints. This approach allows automaticevaluation of each linguistic test point notshown in BLEU and provides intuitive insightinto the strengths and flaws of NMT systems inhandling various important linguistic test points.The linguistic test points used for automaticevaluation were 58 and consisted of 630sentences. We conducted the evaluation of twobidirectional English/Korean NMT systems.BLEUs of English-to-Korean NMT systemswere 0.0898 and 0.2081 respectively, and theirautomatic evaluations by linguistic test pointswere 58.35% and 77.31%, respectively. BLEUsof Korean-to-English NMT systems were0.3939 and 0.4512 respectively, and theirautomatic evaluations by linguistic test pointswere 33.10% and 40.47%, respectively. Thismeans that the automatic evaluation approachby linguistic test points has similar results asBLEU assessment. According to automaticevaluation by linguistic test points, we knowthat both English-to-Korean NMT systems andKorean-to-English NMT systems havestrengths in polysemy translations, but hasflaws in style translations and translations ofsentences with complex syntactic structures.
KSP Keywords
Machine Translation(MT), Neural machine translation, automatic evaluation