Number of words: 876<\/p>\n\n\n\n
The artificial-intelligence industry is often compared to the oil industry: once mined and refined, data, like oil, can be a highly lucrative commodity. Now it seems the metaphor may extend even further. Like its fossil-fuel counterpart, the process of deep learning has an outsize environmental impact.<\/p>\n\n\n\n
In a new paper, researchers at the University of Massachusetts, Amherst, performed a life cycle assessment for training several common large AI models. They found that the process can emit more than 626,000 pounds of carbon dioxide equivalent\u2014nearly five times the lifetime emissions of the average American car (and that includes manufacture of the car itself).<\/p>\n\n\n\n
It\u2019s a jarring quantification of something AI researchers have suspected for a long time. \u201cWhile probably many of us have thought of this in an abstract, vague level, the figures really show the magnitude of the problem,\u201d says Carlos G\u00f3mez-Rodr\u00edguez, a computer scientist at the University of A Coru\u00f1a in Spain, who was not involved in the research. \u201cNeither I nor other researchers I\u2019ve discussed them with thought the environmental impact was that substantial.\u201d<\/p>\n\n\n\n
The paper specifically examines the model training process for natural-language processing (NLP), the subfield of AI that focuses on teaching machines to handle human language. In the last two years, the NLP community has reached several noteworthy performance milestones in machine translation, sentence completion, and other standard benchmarking tasks. OpenAI\u2019s infamous GPT-2 model, as one example, excelled at writing convincing fake news articles.<\/p>\n\n\n\n
But such advances have required training ever larger models on sprawling data sets of sentences scraped from the internet. The approach is computationally expensive\u2014and highly energy intensive.<\/p>\n\n\n\n
The researchers looked at four models in the field that have been responsible for the biggest leaps in performance: the Transformer, ELMo, BERT, and GPT-2. They trained each on a single GPU for up to a day to measure its power draw. They then used the number of training hours listed in the model\u2019s original papers to calculate the total energy consumed over the complete training process. That number was converted into pounds of carbon dioxide equivalent based on the average energy mix in the US, which closely matches the energy mix used by Amazon\u2019s AWS, the largest cloud services provider.<\/p>\n\n\n\n
They found that the computational and environmental costs of training grew proportionally to model size and then exploded when additional tuning steps were used to increase the model\u2019s final accuracy. In particular, they found that a tuning process known as neural architecture search, which tries to optimize a model by incrementally tweaking a neural network\u2019s design through exhaustive trial and error, had extraordinarily high associated costs for little performance benefit. Without it, the most costly model, BERT, had a carbon footprint of roughly 1,400 pounds of carbon dioxide equivalent, close to a round-trip trans-America flight for one person.<\/p>\n\n\n\n
What\u2019s more, the researchers note that the figures should only be considered as baselines. \u201cTraining a single model is the minimum amount of work you can do,\u201d says Emma Strubell, a PhD candidate at the University of Massachusetts, Amherst, and the lead author of the paper. In practice, it\u2019s much more likely that AI researchers would develop a new model from scratch or adapt an existing model to a new data set, either of which can require many more rounds of training and tuning.<\/p>\n\n\n\n
To get a better handle on what the full development pipeline might look like in terms of carbon footprint, Strubell and her colleagues used a model they\u2019d produced in a previous paper as a case study. They found that the process of building and testing a final paper-worthy model required training 4,789 models over a six-month period. Converted to CO2 equivalent, it emitted more than 78,000 pounds and is likely representative of typical work in the field.<\/p>\n\n\n\n
The significance of those figures is colossal\u2014especially when considering the current trends in AI research. \u201cIn general, much of the latest research in AI neglects efficiency, as very large neural networks have been found to be useful for a variety of tasks, and companies and institutions that have abundant access to computational resources can leverage this to obtain a competitive advantage,\u201d G\u00f3mez-Rodr\u00edguez says. \u201cThis kind of analysis needed to be done to raise awareness about the resources being spent […] and will spark a debate.\u201d\u201cWhat probably many of us did not comprehend is the scale of it until we saw these comparisons,\u201d echoed Siva Reddy, a postdoc at Stanford University who was not involved in the research.<\/p>\n\n\n\n
The results underscore another growing problem in AI, too: the sheer intensity of resources now required to produce paper-worthy results has made it increasingly challenging for people working in academia to continue contributing to research.\u201cThis trend toward training huge models on tons of data is not feasible for academics\u2014grad students especially, because we don\u2019t have the computational resources,\u201d says Strubell. \u201cSo there\u2019s an issue of equitable access between researchers in academia versus researchers in industry.\u201dStrubell and her coauthors hope that their colleagues will heed the paper\u2019s findings and help level the playing field by investing in developing more efficient hardware and algorithms. Reddy agrees. \u201cHuman brains can do amazing things with little power consumption,\u201d he says. \u201cThe bigger question is how can we build such machines.\u201d<\/p>\n\n\n\n