How can you efficiently process real-time streaming data? One of our customers was looking for a robust solution to process incoming stream of data from various IoT enabled devices (for example, a Windmill). We needed a highly scalable, real-time streaming data processor and a way to transform this data real-time and store the transformed data in NoSQL Database for further usage and analysis. We designed the solution using AWS Kinesis Firehose (Streaming data Ingestion), AWS Kinesis Analytics (Stream processor) and AWS Lambda (send transformed data to downstream systems such as Database or Analytics engine like QuickSight).
How can you efficiently process real-time streaming data?
Traditionally, we run native SQL queries against static data in a relational DB system where rows get added, deleted and modified with respect to time. The whole paradigm shifts when you have an incoming data stream constantly changing data. AWS Kinesis Analytics allows us to run simple SQL queries on such streaming data. Important fact to consider here is that data keeps getting added at a rapid pace and therefore AWS allows us to run our SQL queries on time segregated chunks of streamed data called as Processing “Windows”.
We decided to build our solution as in below diagram using Kinesis Analytics (transformation of data in transit) and Lambda in Post-processing for sending transformed data to downstream services such as DynamoDB (for further analysis on data at Rest), Amazon S3 (raw data backup), QuickSight (Analytics and Dashboards). Kinesis analytics also supports pre-processing via Lambda Functions in case we want to prepare the incoming data or perform any data enrichment operations to add additional metadata which cannot be added while generating the original Data from IoT devices.
With the help of AWS services, we were able to build a resilient system for predictive and prescriptive maintenance of Wind Mill. Kinesis Stream, Firehose and Kinesis Data Analytics are very powerful services when combined with other services offered by AWS and can transform real-time domains like Air Traffic Control (ATC’s), IoT monitoring and maintenance, Industrial IoT solutions, Smart Home/Smart Societies, Website monitoring and Clickstream generation and processing for analytics.
These services grant us the super power to act upon anomalies and critical situations instantly with a fast response time thereby avoiding casualties or further damage in the system while reducing the maintenance cost with almost zero to Nil downtime.