Organizations are overwhelmed by data streams from products, assets, cloud services, apps & IT infrastructure. Most data is only ephemerally useful, but streams never stop. How can they derive continuous intelligence and automate decisions without a store-then-analyze architecture?
This talk will present an Apache 2.0 licensed platform for continuous intelligence (SwimOS) that uses stateful in-memory processing for continuous analysis, learning and prediction. Swim apps
- Always have an answer: Algorithms have been to be adapted to analyze, train and predict continuously - with computation driven by data.
- Continuously analyze:
- Each event is statefully processed in memory, in real-time, offering 6 orders of magnitude performance win over database accesses.
- Analysis is continuous because data streams are boundless. Insights are necessarily “up to now”, and also form a real-time stream
- Analyze in context: Fluid relationships between real-world data sources - like containment or geospatial proximity, and computed relationships like correlation - are critical for applications that reason about the meaning of events. SwimOS allows algorithms to continuously compute system-wide insights for contextually related 'things', even as those relationships change
The talk will demonstrate an application that processes > 4PB/day of signal data from mobile cell towers to enable the operator to optimize connection quality on-the-fly for over 150M mobile devices.
SwimOS is a set of small extensions to Java reminiscent of the actor model, that uses an application-state cache-coherency protocol called WARP that ensures that concurrent in-memory actors - called web agents - stay in sync, even when distributed. SwimOS uses streaming data to build a scaled-out graph of stateful, concurrent web agents that are analytical “digital twins” of data sources. Each statefully processes raw data from a single source. Agents link to each other based on context discovered in the data, building a graph that reflects real-world relationships. Linked agents see each other’s state changes in memory, in real-time. Agents concurrently compute on their own states and that of others they are linked to. They analyze, learn and predict, and continuously stream enriched insights & responses to users, UIs and enterprise applications.
