Understanding Azure Time Series Insights

What is Azure Time Series Insights?

Azure Time Series Gen2 is an end-to-end Internet of Things (IoT) analytics service designed for industrial IoT deployments. It provides powerful APIs you can use to integrate it with current workflows and applications. 

Azure Time Series Insights collects, processes, stores, queries, and visualizes IoT data at scale, adding context and optimizing it for time series analysis. It is designed to explore data ad hoc and analyze operations, helping you discover hidden trends, spot anomalies, and perform root-cause analysis.

In this article, you will learn:

• What is an Azure Time Series Model?
  • Time Series Model Components
• Azure Time Series Data Storage
  • Storage Tiers and Data Availability
  • Warm Store
  • Cold Store
• Azure Time Series Best Practices
  • Using Warm Store and Cold Store
  • Configure Time Series IDs and Timestamp Properties
  • Optimize Your Events
  • High Availability

What is an Azure Time Series Model?

A Time Series Model is a central entity in Azure Time Series, which lets you manage, maintain, and enhance time series datasets for analysis.

The Time Series Model provides the following capabilities:

• Create and manage calculations and formulas using scalar functions and aggregate operations
• Define hierarchical relationships to enable search, cross referencing, and navigation
• Define properties associated with data instances, and use them to build hierarchies

Time Series Model Components

A Time Series Model has three key components: instances, hierarchies, and types. You use these components to specify a model for analysis of time series data, and organize the data.

Source: Azure

Instances

A Time Series Model instance is an individual time series. It has a unique identifier, the time series ID, which is typically the ID of the asset or device that generated the data.

Each instance can be described by additional properties, such as type, description, name, hierarchies, and instance fields. Instance fields are collections of information such as hierarchy level, device operator, manufacturers, and more.

As soon as you add an event source in Azure Time Series Insights, the system discovers time series and creates Time Series Models. You can update these models or create new ones using Time Series Model queries.

Hierarchies

The Time Series Model organizes instances by specifying attribute names and hierarchical relationships between them. One instance can map to one hierarchy or multiple hierarchies—see an example below.

Source: Azure

Types

Time Series Model types are associated with a specific instance, and allow you to define variables or expressions used to perform calculations.

Types can have one or more variables. For example, one Time Series Model instance could have temperature sensor type, consisting of average temperature, minimum temperature, and maximum temperature variables.

Azure Time Series Data Storage

Azure Time Series Insights works on datasets stored in your Azure Storage account. Here are some key considerations for storing time series datasets in Azure.

Storage Tiers and Data Availability

Azure Time Series Insights Gen2 partitions and indexes data, to optimize query performance. After indexing the data, you can query data from hot storage (if enabled) and cold storage. 

The amount of data captured and the processing speed of each partition affects availability. You can configure alerts to be notified when data processing is lagging in your environment.

Warm Store

You can access data from hot storage through the Time Series Query API, Time Series Insight TSI Explorer, or the Power BI connector. Hot storage queries are free and have no quota, but you can perform up to 30 concurrent requests.

When warm data store is enabled, it works as follows:

• Ingests all data—if warm store is enabled, all data flowing into the environment is routed to it, regardless of event timestamps. The streaming pipeline is built for near-real-time streaming and does not support collection of past events.
• Retention period—calculated based on the date and time the event was indexed in warm storage, not the timestamp of the event.
• No back-filling—if you enable hot storage in an existing environment that already has data in cold storage, the data will not be repopulated into warm storage.

Cold Store

For events sent to cold storage, Azure Time Series Insight Gen2 keeps up to two copies of each event in the Azure Storage account. Events are stored in chronological order. Over time, Azure Time Series Insights Gen2 repartitions your data to optimize high-performance queries. Data is stored in the Azure storage account indefinitely.

Azure Time Series Best Practices

Monitor Azure Time Series Insights

Azure Time Series can be a critical part of IoT data pipelines. It is important to set up monitoring, to ensure the service is working correctly, identify problems and resolve them. You can use Azure Monitor to perform continuous monitoring of Azure Time Series Insights. 

Focus on metrics such as bytes received from all event sources, bytes successfully processed, and bytes available for processing. Also watch the TSIIngress table, which shows errors that occur in the event ingress pipeline.

Using Warm Store and Cold Store

You can choose to enable a “warm store”, which enables faster response times, and provides a retention period of 7 to 30 days. Note that data that needs to be retained for more than 30 days is served from the “cold store”, and incurs a fee for data access. Interactive analytics on recent data should reside in warm storage, while long-term trends and pattern analysis should reside in cold store.

Configure Time Series IDs and Timestamp Properties

In Azure Time Series, you need to select an ID and three keys for each time series, which you can later use to partition the data. You should also designate a timestamp property when adding event sources for future tracking. Otherwise, the event’s enquing time will be used as its timestamp. Also, notice that timestamp values are case-sensitive and should be formatted to individual event source specifications.

Optimize Your Events

Ensure that events are optimized before sending them to Azure Time Series Insights. It is recommended to denormalize events before ingesting them. You should store metadata in your time series model, and ensure instance fields and events only contain vital information, like time series ID and the timestamp property.

High Availability

Time Series Insights leverages region-level redundancies to high availability. You can perform disaster recovery within Azure using Azure Site Recovery (ASR). Other features include geo-replication and load balancing for failover, data recovery, and backup of on-premises or Azure-based VMs using the Azure Backup service.

To ensure your devices and users have global, cross-region high availability, make sure you enable the correct Azure features. 

Conclusion

This article explained about Azure Time Series Insights components, storage, and best practices that can help you derive insights from IoT time series data in the cloud.

FAQ