Human evaluation usage (for linguists)

When an evaluator is invited to participate in an evaluation job of any type, they will receive an email from MT Companion. The email will contain summary information about the job and will offer the invitee the possibility to Accept or Reject the assignment. Selecting either of these options will launch MT Companion in a browser window.

  • If the invitee rejects the assignment, MT Companion will display a message confirming that the job has been rejected. The project manager will be notified and can invite a new evaluator.

  • If the invitee accepts the assignment, MT Companion will display a message confirming that the job has been accepted. The evaluator can then select Start Evaluation to begin performing the evaluation. The project manager will be notified that the job has been accepted.

Once the job has been accepted, the evaluator can access the evaluation job by returning to MT Companion following the Job Info link in the original invitation email.

HEVAL Job List view

The Human Evaluation Job List view is the primary landing page for evaluators. This view displays all assignments in the left column. This list is searchable, sortable, and filterable so the evaluator can organize their work based on their preferences.

When an evaluation job is selected, the details of that job will be displayed in the main view on the right side of the screen. Here, a summary of the job is displayed at the top. In the Language Pairs section, the details are shown, including the Status and Progress of the assignment.

Human Evaluation Job List

HEVAL Job execution (general)

To execute or continue an evaluation job, select the Start/Continue Job button run. When the job is first launched, MT Companion will display a welcome message providing information about the evaluation job, including the name of the job, the number of units in the job and the language pair. After the first launch, the evaluator will be brought directly to the next incomplete segment in the job.

HEVAL Job navigation

While working on a human evaluation assignment, the top bar will provide general information about the current job.

Human Evaluation Top Navigation Bar

  1. Job type. Select this to display general information about the current assignment.

  2. Current segment out of total number of segments in the job.

  3. Countdown displaying the time remaining until the due date.

  4. Screen layout (vertical or horizontal). This option is not available for CA jobs.

  5. Continue Later (for incomplete assignments) or Exit (for completed assignments). Select this to return to the Job List view.

To progress through the segments, the evaluator must perform the evaluation and then select Save and Next. The Previous sentence button can be used to go back and review previously evaluated segments, but these segments will no longer be editable.

The bottom bar will show the time spent on the current segment in the current session and the overall completion percentage for the current job.

Human Evaluation Progress Bar

When the last segment is evaluated, MT Companion will display an Evaluation completed message. The evaluator can optionally provide general feedback on the assignment, and then submit the job (including the feedback).

Human Evaluation Completed message


HEVAL Job Review

Any completed job can be returned to the linguist for review. When the job status is set to In-review, it unlocks the ability to update already- evaluated segments.

The edit mode for the selected segment is opened with the Edit button in the job execution screen in the top navigation bar:

Human Evaluation Edit button

When the Edit button is clicked, the evaluated segment is opened in edit mode, and the user can update all elements of the evaluation.

Updating of the segment can be canceled or saved at any time using the Cancel or Save buttons:

Human Evaluation Edit mode buttons

When the Save button is clicked, all updated values are saved in the database.

Each updated segment is marked in the top-right corner:

Human Evaluation Edit mode buttons

Note

The linguist should be always notified about existing review by the PM and instructed on what has to be reviewed.

Review completed

When all affected segments have been updated, the linguist must mark the job as Reviewed. This is done by clicking the reviewed button, which appears in the list of jobs:

Human Evaluation Edit mode buttons

When clicked, the job is marked as Reviewed and the review is complete for the linguist.

Warning

Setting the status to Reviewed cannot be undone by the linguist! After the button is clicked, the button will disappear from the list of jobs.

Note

The job can be sent for review several times (Reviewed jobs can be set back to In-review state by the PM).


HEVAL Job execution by job type

Each job type has a different goal and interface. See the following sections for guidance on how to perform evaluations for each job type.