COMET score
COMET is a neural translation-quality metric. Instead of counting matching words, it uses a language model to represent the meaning of the source, the output and the reference, then predicts the score a human evaluator would give. It correlates with human judgement far better than word-overlap metrics like BLEU.
How it works
COMET feeds the source, the machine output and a human reference into a neural model that represents each as meaning, not just words, then predicts the quality score a human evaluator would give. Because it works at the level of meaning, it rewards a correct paraphrase that word-overlap metrics would punish.
That is its strength and its catch. It tracks human judgement far more closely than BLEU, but it is itself a model prediction, so it can be wrong, and it is weaker for low-resource languages where the underlying model has seen less.
How SourceTarget uses it
SourceTarget uses COMET as the meaning-aware component of its composite quality score, balancing the surface metrics that only see words. Where COMET and a word-overlap metric disagree sharply, that gap itself flags a segment worth a human's eye.
COMET score compared with BLEU score
| COMET score | BLEU score | |
|---|---|---|
| Measures | Predicted human judgement of meaning | Word overlap with a reference |
| Correlates with humans | Strongly | Weakly, especially on paraphrase |
| Cost and transparency | Heavier, a model prediction | Cheap, transparent, repeatable |
| Weak spot | Low-resource languages | Correct wording it did not expect |