Dalarna University's logo and link to the university's website

This thesis addresses the gap in the machine translation evaluation metrics, particularly in translating complex company reports. Traditional metrics like NIST focus on n-gram informativeness, METEOR on precision, recall, and matching techniques, and ChrF++ on lexical accuracy through n-grams at character and word levels but they fail to capture the full scope of grammatical accuracy and lexical cohesion. The aim of this thesis is to evaluate machine translations of company reports for grammatical and lexical accuracy. To achieve this, a new combined score was introduced which incorporates grammatical and lexical features. Using data from the GRI’s Sustainability Disclosure Dataset, this thesis analyzed translations of reports in English, French, Chinese, and Japanese languages. The findings of the thesis indicate that while traditional metrics offer valuable insights,they lack comprehensive linguistic evaluative capacity. The thesis concludes that enhancing traditional metrics to include deeper linguistic elements is essential for improving translation quality. Future research should explore advanced models like BERT and GPT-4, incorporate human reviews, and expand cross-linguistic validation to enhance translation quality and reliability.