Human evaluation usage (for linguists)

When an evaluator is invited to participate in an evaluation job of any type, they will receive an email from MT Companion. The email will contain summary information about the job and will offer the invitee the possibility to Accept or Reject the assignment. Selecting either of these options will launch MT Companion in a browser window.

  • If the invitee rejects the assignment, MT Companion will display a message confirming that the job has been rejected. The project manager will be notified and can invite a new evaluator.

  • If the invitee accepts the assignment, MT Companion will display a message confirming that the job has been accepted. The evaluator can then select Start Evaluation to begin performing the evaluation. The project manager will be notified that the job has been accepted.

Once the job has been accepted, the evaluator can access the evaluation job by returning to MT Companion following the Job Info link in the original invitation email.

HEVAL Job List view

The Human Evaluation Job List view is the primary landing page for evaluators. This view displays all assignments in the left column. This list is searchable, sortable, and filterable so the evaluator can organize their work based on their preferences.

When an evaluation job is selected, the details of that job will be displayed in the main view on the right side of the screen. Here, a summary of the job is displayed at the top. In the Language Pairs section, the details are shown, including the Status and Progress of the assignment.

Human Evaluation Job List

HEVAL Job execution (general)

To execute or continue an evaluation job, select the Start/Continue Job button run. When the job is first launched, MT Companion will display a welcome message providing information about the evaluation job, including the name of the job, the number of units in the job and the language pair. After the first launch, the evaluator will be brought directly to the next incomplete segment in the job.

HEVAL Job navigation

While working on a human evaluation assignment, the top bar will provide general information about the current job.

Human Evaluation Top Navigation Bar

  1. Job type. Select this to display general information about the current assignment.

  2. Current segment out of total number of segments in the job.

  3. Countdown displaying the time remaining until the due date.

  4. Screen layout (vertical or horizontal). This option is not available for CA jobs.

  5. Continue Later (for incomplete assignments) or Exit (for completed assignments). Select this to return to the Job List view.

To progress through the segments, the evaluator must perform the evaluation and then select Save and Next. The Previous sentence button can be used to go back and review previously evaluated segments, but these segments will no longer be editable.

The bottom bar will show the time spent on the current segment in the current session and the overall completion percentage for the current job.

Human Evaluation Progress Bar

When the last segment is evaluated, MT Companion will display an Evaluation completed message. The evaluator can optionally provide general feedback on the assignment, and then submit the job (including the feedback).

Human Evaluation Progress Bar


HEVAL Job execution by job type

Each job type has a different goal and interface. See the following sections for guidance on how to perform evaluations for each job type.