一句话：提出基于NLP的测试步骤聚类方法，将语义相似的NL测试步骤归并到同一测试API，显著减少人工实现量。
针对大型工业应用中大量自然语言（NL）系统测试用例的自动化，测试人员需为每个测试步骤手工实现测试API，导致语义相似步骤被重复实现。本文提出基于自然语言处理的测试步骤聚类方法，将可映射到同一API的相似步骤归为一类。方法包括领域特定词嵌入训练、基于Relaxed Word Mover's Distance的相似度度量，以及层次凝聚聚类与K-means后精修结合，以得到高质量且可人工调整的聚类。在微信移动应用上的评估表明，该方法聚类准确率高；相比基线，聚类质量提升79.8%，聚类数量减少65.9%，从而降低下游人工实现测试API的工作量。
For large industrial applications, system test cases are still often described in natural language (NL), and their number can reach thousands. Test automation is to automatically execute the test cases. Achieving test automation typically requires substantial manual effort for creating executable test scripts from these NL test cases. In particular, given that each NL test case consists of a sequence of NL test steps, testers first implement a test API method for each test step and then write a test script for invoking these test API methods sequentially for test automation. Across different test cases, multiple test steps can share semantic similarities, supposedly mapped to the same API method. However, due to numerous test steps in various NL forms under manual inspection, testers may not realize those semantically similar test steps and thus waste effort to implement duplicate test API methods for them. To address this issue, in this paper, we propose a new approach based on natural language processing to cluster similar NL test steps together such that the test steps in each cluster can be mapped to the same test API method. Our approach includes domain-specific word embedding training along with measurement based on Relaxed Word Mover’sDistance to analyze the similarity of test steps. Our approach also includes a technique to combine hierarchical agglomerative clustering and K-means clustering post-refinement to derive high-quality and manually-adjustable clustering results. The evaluation results of our approach on a large industrial mobile app, WeChat, show that our approach can cluster the test steps with high accuracy, substantially reducing the number of clusters and thus reducing the downstream manual effort. In particular, compared with the baseline approach, our approach achieves 79.8% improvement on cluster quality, reducing 65.9% number of clusters, i.e., the number of test API methods to be implemented.
DOI 原文 ·
@inproceedings{paperbot5759,
title = {Clustering test steps in natural language toward automating test automation},
author = {Linyi Li and Zhenwen Li and Weijie Zhang and Jun Zhou and Pengcheng Wang and Jing Wu and Guanghua He and Xia Zeng and Yuetang Deng and Tao Xie},
booktitle = {FSE},
year = {2020},
doi = {10.1145/3368089.3417067}
}