Kafka and Kafka Streams have emerged as the de facto standard for scalable, high volume and low-latency real time data processing in the industry. Complex event processing is concerned with the detection of patterns in raw events and their transformation to higher level events that are more business relevant. This talk shows how to implement scalable, fault tolerant and declarative complex event processing based on Kafka Streams.

Target Audience:Data Engineer, Data Scientist, Data Architect
Prerequisites: Basic understanding of real time data processing and Apache Kafka.
Level: Advanced

Extended Abstract:

Kafka and Kafka Streams have emerged as the de facto open source standard for real time, low latency and high throughput data integration in a large variety of industries. Besides providing traditional operations such as grouping, aggregations and table-table joins, Kafka Streams treats streams of data as first class citizens and offers some unique features geared at real time data wrangling such as aggregation of data across different types of time windows, real time data enrichment, and joins between streams of data. While both types of operations have emerged from the ambit of relational databases, the complex event processing community has focused their attention on declarative, pattern based recognition of business relevant events from lower level, raw events. Examples for operations supported by complex event processing are sequences, negation, conjunction, disjunction and repetition. This talk shows how sequences, negation and regular expressions on event streams can be implemented in a scalable and fault tolerant manner based upon the Kafka Streams DSL and the Kafka Streams Processor API. The provided examples are relevant to use cases such as detection of shoplifting, security systems, operations of production lines, and online shopping.