How-to for real-time alerting, analytics and reporting at scale with Apache Kafka and Apache Ignite

How-to for real-time alerting, analytics and reporting at scale with Apache Kafka and Apache Ignite

Are you tasked to build a system or upgrade an existing architecture to a solution capable of handling unbound streams of data, do real-time alerting, storing always growing terabytes and petabytes of data and, finally, act on the data within milliseconds SLAs. Sounds like a challenging but rather a common task? Yes! Should you as an engineer or architect build it up off the ground? Unlikely!

What if we tell you that by integrating Apache Kafka with Apache Ignite you’ll solve all of the requirements faster and easier. A battle-tested recipe is simple - take Kafka Connect and have your data stream through Kafka pipelines, add a pinch of KSQL to act on the streams with SQL in real-time with zero delays, rinse and flush the preprocessed data in Ignite as in-memory databases and get further insights by analyzing your hot and cold datasets.

In this talk, Denis and Viktor will demonstrate how to implement the solution in practice, will explain architectural reasoning and the benefits of real-time integration, share common usage patterns. As always, we'll build a streaming data pipeline using nothing but our bare hands, Apache Ignite, Kafka Connect, and KSQL.




Regency Ballroom B


Product Management
GridGain Systems
Denis Magda runs Product Management at GridGain Systems, and is the Vice President of the Apache Ignite PMC. He is an expert in distributed systems and platforms who actively contributes to Apache Ignite and helps companies and individuals deploy it for mission-critical applications. You can be sure to come across Denis at conferences, workshop and other events sharing his knowledge about use case, best practices, and implementation tips and tricks on how to build efficient applications with in-memory data grids, distributed databases and in-memory computing platforms including Apache Ignite and GridGain.

Before joining GridGain and becoming a part of Apache Ignite community, Denis worked for Oracle where he led the Java ME Embedded Porting Team -- helping bring Java to IoT.
Partner Solutions Architect
Jeff Bean is a Partner Solution Architect at Confluent and an experienced software engineer and technical evangelist. Jeff has dedicated years to open source software, with expertise spanning middleware, big data, event streaming and stream processing. He is currently responsible for the Confluent Verified Integrations Program that helps build and verify integrations and connectors with Confluent Platform and Kafka, including the Ignite connector from Gridgain.

Slides & Recordings

   Download Slides