Analytics play a huge role in gathering insights from large data sets from various sources like Application Logs, User Input and more specifically machinery edge data in an IOT use case. Stored data sets can be processed on demand or at runtime while generating any user-friendly reports or dynamic dashboards. But it becomes challenging when there is large amount of streaming data that needs to be processed / transformed real-time with millisecond or lesser latency. If this latency is not taken care of, we would miss important events in the incoming stream thereby degrading the final analytical significance.
AWS Kinesis Data Analytics Stream Processing
At PROLIM, there was a requirement of a real-time stream processor to collect incoming data with least possible latency, process it and store it in MySQL and Dynamo-DB for further use. We opted for below configuration using AWS Kinesis Analytics as Stream processor and Lambda as destination processor for our Windmill IoT use case.