一句话：从 NVIDIA 工程实践出发，指出大语言模型训练与推理的系统优化是当前最紧迫的计算挑战。
本文是 NVIDIA 的 Bryan Catanzaro 的主旨演讲摘要。演讲以 ChatGPT 的爆发式增长为切入点，指出大语言模型正以根本性的方式被集成进科技产业的产品之中，其文本生成、问答与问题求解能力带来了巨大可能性，但要让这些系统可靠、可信并与应用无缝融合，仍有大量研究要做。演讲的核心关注点是模型背后的计算挑战：由于模型收敛所需的工作量极其庞大，训练与部署系统必须具备极高的可扩展性与极致效率，而当前推理部署成本高昂，已成为落地的障碍，必须显著降低。演讲介绍了 NVIDIA 在优化大语言模型训练与推理系统方面的工作，并展望了仍然存在的开放挑战。
ChatGPT recently became one of the fastest growing new applications in history, thanks to its intriguing text generation capabilities that are able to answer questions, write poetry, and even problem solve. Large Language Models are now being integrated in fundamental ways into products around the tech industry. The possibilities are extraordinary, but much research remains to make these systems reliable and trustworthy, as well as integrate them into applications seamlessly. Additionally, the computational challenges behind large language modeling are also quite important. Systems for training and deploying these models must be highly scalable and run at extreme efficiency, because the amount of work necessary to converge a model can be extraordinarily large. The cost of deploying these models is a barrier to their deployment and must be lowered significantly. In this talk, I’ll discuss the work we have been doing at NVIDIA to optimize systems for Large Language Model training and inference, and highlight some of the challenges that remain for future work.