This project grew in response to reviewer comments on a short article discussing problems in machine translation (MT). One reviewer objected that my criticisms regarding word sense disambiguation were based on “anecdotal evidence” rather than empirical quantitative results. The other reviewer noted “unproven statements”. Frankly, I agreed with those comments. Despite having spent years becoming familiar with the inner workings of Google Translate (GT) to the greatest extent I could as an outsider, I could only base my discussion on a pile of “input X in Language A produces output Y in Language B” examples. Neither I nor any other researcher could make definitive statements about the performance of GT in any given language, much less about its performance across the board, because no research had been published with consistent analysis of GT results.

What followed became a project that subsumed several months, and involved hundreds of people. I sat down to design a survey of GT that would satisfy the reviewers’ cogent objections, producing objective and informative results through consultation with native speakers, in a survey that could be consistently understood and completed across languages. I started with twenty languages where I had close contacts. When those results started rolling in, I realized that the selection was still too scattershot, so I scrolled through my Rolodex and sent another twenty invitations, and then twenty more. At that point, in for a dime, in for a dollar (don’t try that expression in GT in any language), it was off to the races (again, don’t try this with GT) to find native speakers who could evaluate every last language in the GT system.

The study design could be criticized on a variety of methodological grounds, such as the choice of examples that were evaluated, the length of those examples, or the reliance on a single annotator for most languages. I will not claim that the research provides a final answer to the evolving question of GT’s performance. I will, however, assert that the study yielded credible results that are fair to GT and valid to the extent that they reach. A well-funded study with more respondents would give more detailed results, and would score particular languages somewhat higher or lower. However, based on comparisons with scores for languages that have been tested by other methods, I submit that the results for each language fall within the range of acceptable error bars, giving a good indication of the overall and relative performance of the system for each language covered. There is no objective way, though to verify my claim to the accuracy of the findings beyond the judgements of these specific 2040 translations as rated by native respondents.

The paper also includes many instances of GT translations that have not been evaluated in cross-language empirical tests. These examples could be objected to on scientific grounds as anecdotal: why this phrase, why this article, why this language? My only defense is that I have made every effort to be fair, often losing hours chasing tiny nuances in languages I do not speak, so that an accurate depiction of the inner workings of GT would ensue. However, I urge you to use the “anecdotal” examples as springboards for your own investigations, so that you can determine the extent to which my conclusions hold for your languages. If you would like to take the lead in conducting a more detailed systematic evaluation for one or more languages, please contact me to discuss the study design, and we can publish your results on this site.

Four more notes about the way these pages were written:

First, I have written largely in the first person singular. This is to distinguish from the first person plural “we”, which I use when discussing findings that arise from survey respondents. The privacy of survey respondents has been scrupulously protected. Each respondent has received notification of the link to this completed study, along with my gratitude for their participation.

Second, I do not adhere to many conventions of academic articles to keep the text dry and omniscient. This is an exegesis about how MT is really used by real people. People use terms like “suss out” all the time (to cite the objection of one reviewer, who writes, “colloquial language indicates of [sic] lower quality of the paper”), and whether GT can handle such language is entirely germane to the topic at hand. For those who think the writing style is too casual, apologies. The primary objective of this web-book is to unpack GT for its consumers. I also hope to engage the scientific NLP community, with a discussion that combines rigorous research and analysis, but y’all can understand a less stilted rhetorical style, whereas a reader from outside the MT world could not crack an opaque academic treatise on the subject. With the aim of being clear to the greatest number of people interested in GT, I’ll err on the side of approachability.

Third, I ended up writing something much longer than a typical academic study, more at the scope of a thesis than a journal article. Initially the piece started as a single article, then became 3 articles bundled with a common introduction and conclusion, and then it started to resemble a book. Neither fish nor fowl, there was no straight line toward standard academic publication in a timely manner. I am not currently employed in an academic position (though I’m open to interesting offers), and I am not trying to pad my CV in order to get tenure, so the conventional publication routes that force academics through the usual slow-moving gatekeepers were becoming an impediment to getting the research before the public. My objective for publication was to share my findings a way that will be interesting and informative to as many people as possible, while also satisfying the rigors of peer review. I thus settled on an unusual format: this website of interlinked articles, with a section devoted to peer commentary (positive and negative) that I will keep updated as the reviews come in. I have attempted to contact the authors of every article cited in this study to extend an invitation to submit their unredacted thoughts to the Commentary section.

The format can best be described by the term “Web Book” , a method of publication that allows for rich multimedia and the inclusion of expanded information as time goes on. Publishing this work as an expandable document, for example, made possible the addition of the qualitative synopsis section, after a friend described all of the aspects of learning the piano that lay between her daughter’s ability to find the right notes and the point at which you might say her efforts were music. I am confident that the research will hold up to inspection by the pros, although some might find particular points objectionable, and part of the point of the web-book is to hold the discussion in open view rather than burying it in academic back alleys. Converting the piece from a bookish document to a dynamic website has taken extra weeks of work, but I hope that you find the result to be robust and compelling.

The advantages of the Web Book format (or webbook, or, as I’ve settled on for these pages as the smoothest read, “web-book”):

• Multimedia (videos, podcasts, animated images)
• Full and open access to extra-large tables and datasets such as all the empirical data and a head-to-head comparison between GT and my major work at Kamusi
• The ability to revise and extend the discussion as I learn new things (for example, citing and responding to new articles from other researchers) or facts change (for example, Google modifies its system)
• The inclusion of peer commentary from experts in a variety of disciplines related to the research, which I can publish unredacted
• The ability to read and respond to comments from the general public
• Readers can zip to the sections that interest them most, and share the pages they find most relevant
• Not beholden to the time-to-publication or content restrictions of an institutional press
• Open to many more people than would find the work in a dusty academic tome. The purpose of doing all this work is for you to learn from it, read it, and share it, to build a new understanding of the world of machine translation with GT at the reins. I hope that making the work free on the web will be satisfying to you, and I thank you for sharing the link to teachyoubackwards.com every time you hear someone talk about translation in general or Google Translate in particular.

I am especially glad that people can read Teach You Backwards in all of the places where the languages covered in this study are spoken. Normal books reach the people who can pay for them, in the markets where publishers have distribution channels. In practice, the traditional publication paradigm means that most books on academic topics end up selling a few thousand copies, mostly to libraries and mostly in North America and Europe, and no private individual on earth spends the $43 that journal publishers demand, without paying royalties, for firewalled articles like this. In the first two months since its release, Teach You Backwards has already been read by people in places as diverse as Kazakhstan, Mongolia, and Zimbabwe. Since I would not make any money from this project no matter how it is distributed, the reward comes from reaching an audience who do not generally have access to academic research that affects their lives

(As a side note, converting the document from Word to the web has had certain effects on the way I am able to present the content. On the positive side, I have been able to add a lot of multimedia, including videos and extra photos,¹ and the amazing Zotero has automated the way I can share references with you. I have also added relevant hyperlinks to informative or entertaining sites around the web. On the down side, though, writing for the web has eliminated automated numbering for pictures, tables, and figures – that is, every time I add or remove an item of that sort, I would have to go into the html code and manually adjust every subsequent number, everywhere it appears, and then do that again if I make another change. This would be a nutty waste of time, so please forgive the occasional gap in numbering where I may have removed a redundant picture, and the kludge of wedging in new items with decimal designations (such as “Picture 10.1”). I have also eliminated the formal outline structure (Part 1, Section A…) because that would also be pernicious to manage on the web, and have instead given each subsection an anchor that I can link you to from other parts of the site (for instance, sending you to the section on Methodology). Happily, I found a tool that is able to manage footnotes correctly, though the conversion took many hours, and broke Zotero until I excised citations from footnotes.)

Fourth, the articles meander through a lot of territory that will strike some readers as tangential. Some tangents are meant to illustrate particular points in ways that will make sense to a general readership. Other sidetracks are meant to respond to objections from others active in language technology, either comments received in the past or anticipated in response to this presentation. As I have explored beneath the rabbit hole that is Google Translate, I have tried (but probably not succeeded) to say all that needs to be said and no more, following this advice from Alice in Wonderland : “Begin at the beginning and go on till you come to the end: then stop.”

References

Items cited on this page are listed below. You can also view a complete list of references cited in Teach You Backwards, compiled using Zotero, on the Bibliography, Acronyms, and Technical Terms page.