Learning from the Past: Adaptive Parallelism Tuning for Stream Processing Systems

RAIDS Lab Authors

Zhengyi Yang

Director

Details

Research Area

Tags

Resources

Abstract

Distributed stream processing systems rely on the dataflow model to define and execute streaming jobs, organizing computations as Directed Acyclic Graphs (DAGs) of operators. Adjusting the parallelism of these operators is crucial to handling fluctuating workloads efficiently while balancing resource usage and processing performance. However, existing methods often fail to effectively utilize execution histories or fully exploit DAG structures, limiting their ability to identify bottlenecks and determine the optimal parallelism. In this paper, we propose StreamTune, a novel approach for adaptive parallelism tuning in stream processing systems. StreamTune incorporates a pre-training and fine-tuning framework that leverages global knowledge from historical execution data for job-specific parallelism tuning. In the pre-training phase, StreamTune clusters the historical data with Graph Edit Distance and pre-trains a Graph Neural Network-based encoder per cluster to capture the correlation between the operator parallelism, DAG structures, and the identified operator-level bottlenecks. In the online tuning phase, StreamTune iteratively refines operator parallelism recommendations using an operator-level bottleneck prediction model enforced with a monotonic constraint, which aligns with the observed system performance behavior. Evaluation results demonstrate that StreamTune reduces reconfigurations by up to 29.6% and parallelism degrees by up to 30.8% in Apache Flink under a synthetic workload. In Timely Dataflow, StreamTune achieves up to an 83.3% reduction in parallelism degrees while maintaining comparable processing performance under the Nexmark benchmark, when compared to the state-of-the-art methods.

Author Affiliations

@inproceedings{han2025learning,
  title = {Learning from the Past: Adaptive Parallelism Tuning for Stream Processing Systems},
  author = {Han, Yuxing and Chen, Lixiang and Wang, Haoyu and Chen, Zhanghao and Zhang, Yifan and Yang, Chengcheng and Hao, Kongzhang and Yang, Zhengyi},
  url = {http://dx.doi.org/10.1109/ICDE65448.2025.00264},
  doi = {10.1109/icde65448.2025.00264},
  booktitle = {2025 IEEE 41st International Conference on Data Engineering (ICDE)},
  publisher = {IEEE},
  year = {2025},
  month = May,
  pages = {3535-3548}
}