一句话：提出 AutoDW，用 LLM 实现端到端自动数据整理并生成可复现源码。
数据整理是将原始数据转换为适合下游任务格式的关键但费力的过程，传统方法耗时、资源密集且易错。本文提出 AutoDW，一个端到端自动数据整理方案，利用大语言模型提升数据准备的自动化与智能化。其特点包括：最小化人工干预的全面自动化、集成 LLM 实现高级数据处理、为整个整理过程生成源代码以保证透明与可复现。通过性能评估，作者展示 AutoDW 在数据整理上的有效性，并分享工业部署经验与教训，表明其在效率、准确性和灵活性上优于现有工具，有潜力改变自动数据准备的格局。
Data wrangling is a critical yet often labor-intensive process, essential for transforming raw data into formats suitable for downstream tasks such as machine learning or data analysis. Traditional data wrangling methods can be time-consuming, resource-intensive, and prone to errors, limiting the efficiency and effectiveness of subsequent downstream tasks. In this paper, we introduce AutoDW: an end-to-end solution for automatic data wrangling that leverages the power of Large Language Models (LLMs) to enhance automation and intelligence in data preparation. AutoDW distinguishes itself through several innovative features, including comprehensive automation that minimizes human intervention, the integration of LLMs to enable advanced data processing capabilities, and the generation of source code for the entire wrangling process, ensuring transparency and reproducibility. These advancements position AuoDW as a superior alternative to existing data wrangling tools, offering significant improvements in efficiency, accuracy, and flexibility. Through detailed performance evaluations, we demonstrate the effectiveness of AutoDW for data wrangling. We also discuss our experience and lessons learned from the industrial deployment of AutoDW, showcasing its potential to transform the landscape of automated data preparation.
@inproceedings{paperbot8355,
title = {AutoDW: Automatic Data Wrangling Leveraging Large Language Models},
author = {Lei Liu and So Hasegawa and Shailaja Keyur Sampat and Maria Xenochristou and Wei-Peng Chen and Takashi Kato and Taisei Kakibuchi and Tatsuya Asai},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695267}
}