一句话:设计并实现兼具分布式执行与可扩展性的流处理语言 SPL,成为 IBM Streams 平台的入口。
大数据分为静态数据与动态数据,后者需要流处理。面向大数据的流分析通常要求分布式系统提供的规模,需在多主机与多核上并行。IBM Streams 是这类分布式流处理系统之一,其早期客户经验表明,可扩展性同样是核心需求:客户希望为流应用构建高性能的领域特定算子。基于分布与可扩展这两大需求,作者设计并实现了 Streams Processing Language(SPL)。本文重点介绍 SPL 的语言设计、分布式运行时与可扩展机制。SPL 现已成为 IBM Streams 平台的入口,被客户用于广泛应用领域的流处理。
Big data is revolutionizing how all sectors of our economy do business, including telecommunication, transportation, medical, and finance. Big data comes in two flavors: data at rest and data in motion. Processing data in motion is stream processing . Stream processing for big data analytics often requires scale that can only be delivered by a distributed system, exploiting parallelism on many hosts and many cores. One such distributed stream processing system is IBM Streams. Early customer experience with IBM Streams uncovered that another core requirement is extensibility, since customers want to build high-performance domain-specific operators for use in their streaming applications. Based on these two core requirements of distribution and extensibility, we designed and implemented the Streams Processing Language (SPL). This article describes SPL with an emphasis on the language design, distributed runtime, and extensibility mechanism. SPL is now the gateway for the IBM Streams platform, used by our customers for stream processing in a broad range of application domains.