CONFLuEnCE: CONtinuous workFLows ExeCution Engine

Data streams have become pervasive and data production rates are increasing exponentially, driven by advances in technology, for example the proliferation of sensors, smart phones, and their applications. This fact effectuates an unprecedented opportunity to build real-time monitoring and analytics applications, which when used collaboratively and interactively, will provide insights to every aspect of our environment, both in the business and scientific domains.

We have identified the need for workflow management systems which are capable of orchestrating the processing of multiple heterogeneous data streams, while enabling their users to interact collaboratively with the workflows in real-time. Based on that model we have built our prototype workflow enactments system: CONFLuEnCE (CONtinuous workFLow ExeCution Engine). CONFLuEnCE is built on top of Kepler, an existing workflow management system, by fusing stream semantics and stream processing methods as another computational domain. Furthermore, we have designed and implemented real-life business and scientific continuous workflow monitoring applications, which attest to the ease of use and applicability of our system.