Real Time Analytics and Stream Processing: An Introduction

What is Real Time Analytics?

Real-time analytics makes it possible for organizations to capture live streams of data, process them very quickly, and extract insights or perform operations on the data in real time or near-real-time. It is based on stream processing technology that can handle a very high throughput of event data. 

There are two types of real time analytics:

• On-demand analytics—providing data or computation results to users or applications in real time. For example, displaying the current product price to a user on an eCommerce site.
• Continuous analytics—processing events on a continuous basis and streaming the results to end users, applications, or a data store. For example, showing live stock market data on a dashboard in a financial institution. 

Real-time analytics has many uses in the digital economy. For example, it can help businesses track customer data and respond with personalized offers, improving customer engagement. It  can enable rapid, automated response to shifts in the market, enabling dynamic pricing on eCommerce sites. Another common use is to process massive volumes of log or sensor data, from IT systems or internet of things (IoT) devices, and using them to drive business decisions.

In this article, you will learn:

• What are Stream Processing Frameworks?
• Top Stream Processing Frameworks
  • Amazon Kinesis
  • Azure Stream Analytics
  • Apache Spark
• Best Practices for Real Time Analytics

What are Stream Processing Frameworks?

Real-time analytics is based on stream processing technology. Stream processing is used to query a continuous stream of data, and quickly process events within that stream, in a short period of time after data is received. Detection time varies from milliseconds to minutes.

Stream processing frameworks provide a stream processor that lets you write business logic for each “actor”—an agent that accepts a stream of events, processes them and generates new events. You can create multiple actors, connect them to a stream, and wire them up to data sources. Events can be sent directly to the stream processor or through a proxy.

The event stream processor is responsible for collecting data, passing it to each actor, making sure it runs in the correct order, collecting the results, and handling errors. Modern stream processors can scale up to very high loads, and can handle millions of events in a single node. 

Top Stream Processing Frameworks

There is a wide variety of stream processing frameworks. Below is a brief overview of the capabilities provided by three popular frameworks.

Amazon Kinesis

Amazon Kinesis is a managed service that simplifies management of real-time data streams. You can use Kinesis to quickly process, collect, and analyze data in real time. Kinesis can handle a variety of data types, including audio and video, application logs, IoT telemetry data, machine learning analytics, and website click flow. Kinesis is often used in combination with AWS Lambda, where real time events from Kinesis are processed by serverless functions.

Azure Stream Analytics

Azure Stream Analytics is a serverless engine that is fully managed by Microsoft. The service offers real-time analysis on multiple data streams. You can integrate with many types of sources, including web applications, social media channels, sensors, and more. However, the inputted data must be in a compatible format, such as JSON, CSV, and AVRO. Additionally, the application logic must be programmed using a query language such as SQL.

Apache Spark

Apache Spark is an open source batch and stream processing framework. The ability to use different types of streams make Spark ideal for machine learning analytics. However, to effectively manage machine learning streams you need to integrate with a distributed storage system and a cluster manager. Spark supports a diverse range of languages, including R, Python, Scala, and Java. 

Best Practices for Real Time Analytics

Once you have a stream processing framework in place, here are some best practices that can help you successfully implement real time analytics in your organization:

Use continuous intelligence for situation awareness

Continuous intelligence enables the system to listen to events in real-time and detect threats and risks as they occur. The system then notifies administrators, proactively pushing alerts via various channels, such as push notifications and email alerts. Some systems can initiate automated response until admins are available. 

Provide multiple personalized views, using a common operating picture

A personalized dashboard helps different system users gain the information relevant to their field. However, to ensure all stakeholders are using the same information, it is critical to provide access to the same real-time analytics feeds.

Focus on the metrics that matter most

Storage costs for continuous data monitoring can quickly accumulate into a high sum. You can prevent overhead by carefully choosing the metrics that matter most to your organization. Start by identifying relevant KPIs, choosing what impacts the performance of your workloads and the user experience. Continue optimizing the metrics and remove irrelevant and redundant data.

Conclusion

This article provided an overview of real-time analytics and stream processing. It explained the differences between On-Demand Analytics and Continuous Analytics, detailed the three most used stream processing frameworks, and highlighted real-time analytics’s best practices.

FAQ