Natural Language – Which Metric to Use for Language Translation

Question

So I am using a pre-trained model to do the language translation

Eg:

Input  = "Good morning"

Output = "Bonjour"

I would like to see if the translation is correct. I do not have any reference sentence for the translated output.

Can someone please tell me how can I compare the input and output and know if the translation is right.

Basically my project is on language translation. If I pass a sentence (any indian language) it should translate the sentence into a indian language I mention so how can I compare them.

Thankyou!

Answer 1

Want to improve this post? Provide detailed answers to this question, including citations and an explanation of why your answer is correct. Answers without enough detail may be edited or deleted.

The task you are looking for is called machine translation quality estimation or sometimes reference-free evaluation (there is a nuanced difference in evaluation, cf. Results of the WMT21 Metrics Shared Task). You can search for tools for that.

If you work with languages that appear in the annual WMT competition, the best choice is the COMET score (paper). It is based on the XLM-R model, so it might work reasonably well also for other languages. Alternatively, BERTScore (paper) based on mBERT or XLM-R might also get reasonable results.