一句话:提出“实证计算”概念,主张用提示让 LLM 直接求解问题实例,并呼吁软件工程建立其分析基础。
LLM 可在不依赖问题计算复杂度的运行时内求解计算问题:无需用编程精确指定算法,而是用提示让 LLM 或 agent 求解实例,输出从分布中采样而非过程式生成。这篇愿景论文探讨这种新计算形式的挑战与机遇,认为经典理性主义计算框架不足以理解其能力与极限。作者呼吁软件工程社区发展分析与改进“实证计算”属性的基础和技术,并提出开放问题:如何分析并提升 LLM 求解计算问题在一般、问题特定与实例特定层面的正确性?实证计算有哪些性质和基本极限?论文旨在将实证计算确立为软件工程中一个及时且问题丰富的研究领域。
Large Language Models (LLMs) can solve any computational problem without an algorithm in a runtime independent of the computational complexity of that problem. Instead of specifying precisely how to solve problem instance using programming, we ask an LLM or agent to solve the problem instance using prompting. Outputs are sampled from a distribution rather than generated procedurally. In this vision paper, we explore the challenges and opportunities of this new form of computation and observe that its capabilities and limits cannot be understood within the classic, rationalist framework of computation. Hence, we appeal to the software engineering (SE) community to develop the foundations and techniques required to analyze the properties of this "empirical computation" as it generates solutions to computational problems: How can we analyze and improve the correctness of LLMs solving a computational problem in the general, in the problem-specific, or in the instance-specific? What are the properties and fundamental limits of empirical computation? This paper aims to establish empirical computation as a field in SE that is timely and rich with interesting problems.