top of page

Translation Quality Research Project

Multidimensional Quality Metrics (MQM)

OutProject Description

 

  1. Assist in the examination of MQM, its practicality, and its possible implementation as a translation standard by:

    • Serving as a liaison for main figures in MQM (Alan Melby, Paul Fields, Arle Lommel) and volunteer translation reviewers.

    • Organizing efforts by volunteer translation reviewers through training, completing the translation assessment using the MQM scorecard, and reporting scorecard results.

    • Participating as a volunteer translation reviewer.

  2. Explore how scorecard technology can enhance translation quality assessment.

  3. Explore possible effects of MQM as a translation standard on the improvement of machine translation.

 

Learning Outcomes

 

This project explores two translation technologies and how a) one of those technologies (scorecard) might affect the future of translation quality assessment and b) how new translation quality metrics might affect the future of the other (machine translation). 

Phases

1. Planning

 

2. Trainning

 

3. Translation Revision

 

4. Alalysis

 

5. Post-Mortem

Planning Phase

Starting with Alan Melby’s presentation in the LING 480 class, the planning phase was very straight forward. With little effort, we were able to get 10 volunteers for translation review using the scorecard. The phases for the project seemed clear to all contributors to the project. However, It turned out that the phases, although seemingly clear to all, were not properly defined which resulted in much waiting and extra work. This may have contributed to a loss of volunteer participation as well.

Trainning Phase

Preparation was completed late due to the miscommunication during the Planning phase. Students were ready to be trained and get to work as of October 29 but training tools and Scorecard accounts had not been set up. Initial training did not start until November 7. Arle provided slides on MQM to the students and then held two separate video conferences to train the volunteers on how to use the Scorecard.

 

On October 31, it was brought to attention that the Basque, Bulgarian, Czech, Dutch, German, and Portuguese languages were not covered by the current volunteers. It was suggested that the LING 480 volunteers find other volunteers of these languages and train them as part of the project. This idea was abandoned but volunteers of these languages were still sought out. Volunteers for Czech, Dutch, German, and Portuguese were found but it was decided that student volunteers for these languages were no longer necessary for this project.

 

On November 10, volunteers were provided with a Google spreadsheet in order to triage 1001 strings of translated text. This task had not been explained during the Planning phase or the training but was carried out by all volunteers excluding one.

Translation Revision Phase

After completing triage of the 1000 strings, each volunteer was given a domain on the gevterm Scorecard website to do the translation assessment. The assessment was meant to include all “almost good” translations but no more than 30% of the total strings from triage were included

on any one domain. As far as I know, all the students who completed the triage also completed this task.

 

Reporting was not necessary for either triage or MQM Scorecard due to performing triage on a Google spreadsheet, to which Alan and Arle both had access to, and the Scorecard report being openly available on gevterm.

 

On November 14, Alan contacted me about creating a set of written instructions for the Scorecard to be published within the tool on gevterm. With help from Tyler Snow, I was able to create these written instructions within a few hours.

Analysis Phase

Analysis was based on the results from the Google spreadsheet and the report from MQM Scorecard. The reports consist of the annotations from each volunteer. In order to evaluate the usefulness of MQM and the Scorecard, the usefulness and quality of the reports have been compared to the time needed in order to obtain them.

Post-Mortem

A five minute video presentation of the report was sent to Adam Wooten via Learningsuite.

Conclusion

MQM provides a clear definition of translation quality and a precise way of assessing translations. Because of the specific data that can be derived from use of MQM, this metric could have an influence for progress in understanding and improving the translation process. As the metric now stands, there would be too many time difficulties for it to be widely implemented in the translation and localization industry. However, a large time investment into the investigation of MQM could lead to advances in automated translation capabilities.

 

To read the full report click here

bottom of page