How translation quality is actually measured

Ask a translation provider how good their work is and you'll usually get one of two answers. The first is an adjective: accurate, professional, high-quality. The second is a number with no explanation attached: 94%, or 87 out of 100. Neither answer tells you anything you can act on, because the adjective isn't a measurement and the number, without knowing what produced it, isn't one either.

Translation quality can be measured. The machine translation research community has spent more than two decades building metrics for it, and those metrics run inside every serious translation platform today. This article explains the main ones in plain English: what each metric actually checks, where each one breaks, and how to read a composite score when a platform hands you one.

You don't need any of the mathematics. You do need to know what question each number is answering, because they're answering different questions.

Why "it reads well" isn't a measurement

Human review is the oldest quality method and it still anchors everything else. But on its own, a reviewer's judgment has two problems at scale. It doesn't repeat: two qualified reviewers routinely disagree about the same passage, and the same reviewer scores differently on Friday afternoon than Monday morning. And it doesn't scale: nobody is human-reviewing 4 million words of website content into 12 languages sentence by sentence, on budget, every time the site changes.

Automatic metrics exist to make quality repeatable and countable. Every one of them works by comparing the machine's output against something, and the whole field divides on what that something is.

What is a BLEU score?

BLEU (bilingual evaluation understudy) is the industry's oldest widely used metric, introduced by IBM researchers in 2002. It compares a machine translation against one or more human reference translations of the same text and counts overlapping word sequences. More overlap, higher score, usually expressed from 0 to 100.

BLEU's virtue is that it's cheap and repeatable, which is why 20 years of research papers report it and why it still serves as a common benchmark when comparing engines. Its flaw is the assumption underneath it: that matching the reference's words means matching its meaning. A translation can say exactly the right thing in different words and score poorly. It can also match many of the words while mangling the meaning and score respectably. For languages with rich morphology or flexible word order, the word-matching assumption gets shakier still.

Treat BLEU as a comparative instrument. A BLEU of 46 versus 38 on the same test set tells you engine A outperformed engine B. A BLEU of 46 in isolation tells you almost nothing about whether a given sentence is fit to publish.

What do chrF and TER add?

Two refinements answer specific weaknesses:

• chrF compares character sequences rather than whole words. That makes it fairer to languages where a single word carries what English spreads across several, and more forgiving of inflection.
• TER (translation edit rate) measures how many edits a human would need to turn the machine output into the reference. It maps directly onto a commercial question: how much post-editing effort is left in this output? Lower is better.

Both are still reference-matching metrics. They refine how the comparison is counted, not what's being compared.

What is COMET, and why did it change things?

COMET represents the newer generation: neural metrics trained on human judgment data. Instead of counting matching words, COMET uses a language model to represent the meaning of the source text, the machine output and the reference, then predicts the score a human evaluator would give. It was trained on years of human quality ratings collected through the machine translation research community's annual evaluation campaigns.

The practical difference: COMET can recognise that "the physician recommended a screening" and "the doctor advised a check-up" are close in meaning, where BLEU sees almost no overlap. Across the research community's benchmark studies, neural metrics like COMET correlate with human judgment substantially better than BLEU does.

The trade-off is interpretability. BLEU's number means something mechanical you can explain in a sentence. COMET's number is a model's prediction, and like all model predictions it inherits the strengths and blind spots of its training data. It's most reliable in well-resourced language pairs and less proven in exactly the community languages where Australian organisations often need it most. That gap matters for a platform serving Punjabi, Nepali or Hazaragi content, and it's one reason no single metric can be trusted alone.

Why doesn't one metric settle it?

Because "quality" isn't one property. A legal document and a health campaign fail in different ways:

• A contract translated with perfect fluency and one wrong term is a bad translation. Terminological accuracy dominates.
• A vaccination campaign translated with perfect terminology that reads like a machine wrote it is also a bad translation. Nobody trusts it, nobody acts on it. Fluency and naturalness dominate.
• A subtitle file can paraphrase freely but must fit timing constraints. A financial disclosure can't paraphrase at all.

Each metric emphasises a different slice of this. BLEU and chrF reward surface fidelity. TER approximates editing effort. COMET approximates human judgment of adequacy and fluency. Run only one of them and you've silently decided which kind of failure you're willing not to see.

How the SourceTarget Quality Score works

This is the reasoning behind the SourceTarget Quality Score. The platform calculates the industry metrics independently on every translation, then combines them into a single 0 to 100 score, with the weighting set by content type. Legal and certified documents weight terminological accuracy more heavily. Marketing and campaign content weights fluency more heavily. The weights follow the failure mode that matters for that content.

Two design choices matter more than the arithmetic:

1. The score arrives with an explanation, not just a number. "82/100. Terminology: high. Fluency: moderate. Three segments flagged for ambiguity in the source text." The flagged segments are the point: a score's job is to direct attention to the sentences a human should look at.
2. The score is a routing instrument. On the platform, the score and the content type together drive what happens next: publish as-is, route to expert review, or escalate to community review for content where cultural fit carries the risk. The score decides where human attention goes. It doesn't replace it.

That's the honest use of automatic metrics in 2026. They're excellent at triage and comparison, unreliable as a final verdict, and the systems that work treat them accordingly.

What should a buyer do with all this?

Four questions to put to any provider or platform quoting you a quality number:

1. What produced this number? If the answer is one metric, ask which failure modes it can't see. If the answer is "our internal score" with no composition, the number is marketing.
2. What was it measured against? Reference-based metrics need reference translations. Ask where those came from and whether they reflect your content type.
3. What happens when the score is low? The useful answer names a process: which segments get flagged, who reviews them, what the reviewer's credential is.
4. Does the score change what you do, or just what you report? A score that routes work to the right level of human review earns its place. A score that only appears on the certificate at the end doesn't.

A council publishing an immunisation campaign, a bank localising a product disclosure and an NFP translating a family violence resource are buying different risk profiles. The measurement should reflect that, and now you know enough to check whether it does.