Quality Assessment (QA) Job execution
The goal of the Quality Assessment job type is to evaluate the suitability of the MT output of a single model. The project manager may select one of two different workflows for this evaluation type:
One-step evaluation
To perform the one-step Quality Assessment type, the evaluator will be presented with a job view that will display the Original Text (source) and the Target Text (MT output). The MT output should be evaluated using the scoring system selected by the project manager.
In the example screenshot below, a 5-point scoring system has been selected.
To evaluate the segment, select the most suitable option from the list provided. When an option is selected, the corresponding Selected Score will appear on the page and the evaluator can proceed to the next segment.
Two-step evaluation
To perform the two-step Quality Assessment type (also known as Quality Assessment with hidden source), the evaluator will be presented with a two-step evaluation process where the job view will first display the Target Text for evaluation without the Original Text for reference.
The MT output should be evaluated using the scoring system selected by the project manager. This segment should be evaluated by selecting the most suitable option from the list provided. When an option is selected, the corresponding Selected Score will appear on the page and the evaluator can proceed to the second step.
The second step will display both the Original Text and the Target Text, along with the score selected in the previous step. This second step gives the linguist the opportunity to revise the score provided in step one. When the score is final selected, the corresponding Selected Score will appear on the page and the evaluator can proceed to the next segment.
Reporting issues for a QA Job
If there is an issue with the Original Text, the evaluator can provide feedback in two ways:
Report Issue section:
For issues with the Original Text (aka. source text), the evaluator can tick one of the boxes in the Or Report Issue section.
The Linguistically Bad Source issue indicates that the source contains segmentation issues, spelling issues, or grammar issues.
The Offensive Source issue indicates that the source text may cause offense or discomfort to the reader.
The Specialist Source issue indicates that the source is domain-specific and therefore cannot be validated by a non-specialist.
Hover over the issue reporting options with your cursor for more information on any of the issues listed.
When any of the issue boxes is ticked, the segment is locked and a score cannot be selected. The evaluator can proceed by selecting Next Sentence.
Additional Feedback
If there is some other issue, the evaluator can leave a comment indicating the issue by selecting the Feedback button. A Feedback form will appear. When feedback has been added, the Feedback button text will be highlighted yellow.