一句话：提出语言模型编程（LMP）及查询语言 LMQL，把提示、脚本与输出约束统一，并编译为高效推理流程。
大语言模型可通过提示完成问答、代码生成等任务，但要让模型达到最优性能或适配特定任务，往往需要实现复杂的任务与模型相关程序，并可能依赖临时性的交互，现有高级 API 难以表达其中的约束与控制流。本文提出语言模型编程（LMP）：将提示从纯文本推广为文本提示与脚本的直观组合，并允许对语言模型输出指定约束，从而在抽象模型内部细节的同时提供高层语义。作者实现了 LMQL（Language Model Query Language），它利用 LMP 提示中的约束与控制流生成高效推理过程，尽量减少对底层语言模型的昂贵调用。评估表明，LMQL 能以直观方式表达多种前沿提示方法，支持现有 API 难以实现的交互流程，在多个下游任务上保持或提升准确率，并在按次付费 API 上节省 26–85% 的成本。
Large language models have demonstrated outstanding performance on a wide range of tasks such as question answering and code generation. On a high level, given an input, a language model can be used to automatically complete the sequence in a statistically-likely way. Based on this, users prompt these models with language instructions or examples, to implement a variety of downstream tasks. Advanced prompting methods can even imply interaction between the language model, a user, and external tools such as calculators. However, to obtain state-of-the-art performance or adapt language models for specific tasks, complex task- and model-specific programs have to be implemented, which may still require ad-hoc interaction. Based on this, we present the novel idea of Language Model Programming (LMP). LMP generalizes language model prompting from pure text prompts to an intuitive combination of text prompting and scripting. Additionally, LMP allows constraints to be specified over the language model output. This enables easy adaption to many tasks while abstracting language model internals and providing high-level semantics. To enable LMP, we implement LMQL (short for Language Model Query Language), which leverages the constraints and control flow from an LMP prompt to generate an efficient inference procedure that minimizes the number of expensive calls to the underlying language model. We show that LMQL can capture a wide range of state-of-the-art prompting methods in an intuitive way, especially facilitating interactive flows that are challenging to implement with existing high-level APIs. Our evaluation shows that we retain or increase the accuracy on several downstream tasks, while also significantly reducing the required amount of computation or cost in the case of pay-to-use APIs (26-85% cost savings).
@article{paperbot1963,
title = {Prompting Is Programming: A Query Language for Large Language Models},
author = {Luca Beurer-Kellner and Marc Fischer and Martin Vechev},
journal = {Proceedings of the ACM on Programming Languages},
volume = {7},
number = {PLDI},
year = {2023},
doi = {10.1145/3591300}
}