New Comparative Assessment With Annotation (CAA) Job
The project manager must fill out the CAA job type form.
FIELD NAME |
DESCRIPTION |
|
|---|---|---|
Name |
A unique and meaningful name for internal tracking. |
Required |
Min. evaluators |
A job must have a minimum of one evaluator. More evaluators can be added later in the project, if necessary. |
Required |
End Date |
Deadline for the evaluators. The end date can be modified later on. |
Required |
Scoring System |
The Scoring System dropdown will only show scoring systems that are compatible with the selected job type. The scoring system can be changed until one or more evaluators has accepted the job. See more information about HEVAL Input File Naming Convention |
Required |
Annotation Set |
The Annotation Set dropdown will only show annotation sets that are compatible with the selected job type or scoring system. The annotation set can be changed until one or more evaluators has accepted the job. See more information about HEVAL Input File Naming Convention |
Optional |
Randomize Record Order |
When this box is ticked, the segments are shown to the evaluators out of order to avoid any bias in the evaluation. |
Optional |
Shuffle Translation Cards |
When this box is ticked, the translation 1 and translation 2 are randomly displayed on the left or right side of the evaluation form. To be used when there is a risc of bias. |
Optional |
Pre-filled Scores |
When this box is ticked, the scores must be included in the input CSV file. It will be pre-filled on the Evaluation screen. |
Optional |
Enable Tie Breaker |
Available only when Pre-filled Scores is not chosen. If enabled, the additional user input is required to confirm if the better score is on translation 1 or 2 or equal during Evaluation Job Execution. |
Optional |
Description |
This optional field allows the project manager to add some additional meta data about the project. |
Optional |
CAA input data format
The input data format requirements for Comparative Assessment jobs are as follows:
File format: CSV
Separator: Semicolon (;)
Minimum number of segments: 2
Maximum number of segments: 2000
Column Name |
Description |
Mandatory |
Data Validation |
|---|---|---|---|
Source |
The source text |
Yes |
Non empty text |
MT1 |
The output from MT model 1 |
Yes |
Non empty text |
Score1 |
The Score for MT1 |
Yes if the job has “Pre-filled Scores Required” option enabled. |
Non empty text |
MT2 |
The output from MT model 2 |
Yes |
No |
Score2 |
The Score for MT2 |
Yes if the job has “Pre-filled Scores Required” option enabled. |
No |
MT1_model_name |
Name of MT model 1 |
No |
No |
MT2_model_name |
Name of MT model 2 |
No |
No |
metadata |
Internal metadata for tracking and data analysis |
No |
No |
Additional_information |
Internal metadata for tracking and data analysis |
No |
No |
Original_Id |
Internal metadata for tracking Original data Id. |
No |
Maximal length 64 characters |
Domain |
Internal metadata for tracking domain category. |
No |
Maximal length 64 characters |