Apache Ignite — Using a Memory Grid for Distributed Computation Frameworks (Spark and Containerized Apps)

While our engineering team was working on a new data cleansing and enrichment Accelerator, we needed a solution that would allow efficient data transmission between elements of the data pipeline. At the same time, we wanted to avoid writing to disk or writing to a database in between discreet computational steps. Finally, we needed to easily pass computational results between loosely coupled computational modules such as Spark and Containerized Apps.

Enter Apache Ignite which provided our team an extremely rich memory-centric, distributed platform with a significant number of feature-rich capabilities including co-located processing, distributed SQL and distributed key value, durable in-memory, plus scalability, availability, and consistency.

This talk with focus on using Apache Ignite and how we use it with both Spark and Containerized Apps plus we will describe how we efficiently shared data across Spark jobs and even across heterogeneous computational frameworks and how doing so benefitted our development and accelerated our solution.

Schedule

Tue, 10/02/2018 - 16:00

Room

Ballroom B

Tracks:

Architecture

Speakers

Chris

Herrera

Chief Innovation Officer

at

Hashmap, Inc.

As Chief Innovation Officer at Hashmap, Chris leads Engineering and Innovation connecting technical challenges across industries to solutions. Prior to joining Hashmap in early 2017, Chris spent 13 years at Schlumberger in a variety of roles including Program Manager, Architect, Real Time Operations Manager, and MWD Engineer.

Chris is an innovation driven technologist with experience in developing distributed and fault tolerant data ingestion systems, stream computing systems, and scalable data storage systems. Chris has hands on experience implementing systems using Apache Kafka, NiFi, Ignite, Spark, Storm, HBase, Cassandra, Ignite and MongoDB. Additionally, he has had hands on experience with a variety of first party cloud solutions. Additional experience designing traditional data management solutions using Oracle, SQLServer, and Tibco EMS. Practical knowledge In Kubernetes and implementing serverless platforms. He is known for practical designs, guided by real-world problems to drive insight and actions from large amounts of data streaming and at rest. Working in teams large and small, Chris has used his technical leadership and collaboration abilities to lead global efforts to deliver products and services of the highest quality.