DCS: A Policy Framework for the Detection of Correlated Data Streams
DocUID: 2019-012 Full Text:
PDF Author: Rakan Alseghayer, Daniel Petrov, Panos K. Chrysanthis, Mohamed Sharaf, Alexandros Labrinidis
Abstract: There is an increasing demand for real-time analysis of large volumes of data streams that are produced at high velocity. The most recent data needs to be processed within a specified delay target in order for the analysis to lead to actionable result. To this end, in this paper, we present an effective solution for detecting the correlation of such data streams within a micro-batch of a fixed time interval. Our solution, coined DCS, for Detection of Correlated Data Streams, combines (1) incremental sliding-window computation of aggregates, to avoid unnecessary re-computations, (2) intelligent scheduling of computation steps and operations, driven by a utility function within a micro-batch, and (3) an exploration policy that tunes the utility function. Specifically, we propose nine policies that explore correlated pairs of live data streams across consecutive micro-batches. Our experimental evaluation on a real world dataset shows that some policies are more suitable to identifying high numbers of correlated pairs of live data streams, already known from previous micro-batches, while others are more suitable to identifying previously unseen pairs of live data streams across consecutive micro-batches.
Published In: Lecture Notes in Business Information Processing, Volume 337
ISBN: 978-3-030-24124-7
Pages: 191-210
Year Published: 2019
Note: (BIRTE (Revised Selected Papers) 2017)
DOI: 10.1007/978-3-030-24124-7_12
Project: DCS Framework Subject Area: Data Streams, Data Exploration
Publication Type: Conference Paper
Sponsor: NSF CBET-1609120, NIH U01HL137159
Similar Publications: