Kobi Bohbot

Understanding Azure Time Series Insights

Understanding Azure Time Series Insights
azure

What is Azure Time Series Insights?

Azure Time Series Gen2 is an end-to-end Internet of Things (IoT) analytics service designed for industrial IoT deployments. It provides powerful APIs you can use to integrate it with current workflows and applications. 

Azure Time Series Insights collects, processes, stores, queries, and visualizes IoT data at scale, adding context and optimizing it for time series analysis. It is designed to explore data ad hoc and analyze operations, helping you discover hidden trends, spot anomalies, and perform root-cause analysis.

In this article, you will learn:

  • What is an Azure Time Series Model?
    • Time Series Model Components
  • Azure Time Series Data Storage
    • Storage Tiers and Data Availability
    • Warm Store
    • Cold Store
  • Azure Time Series Best Practices
    • Using Warm Store and Cold Store
    • Configure Time Series IDs and Timestamp Properties
    • Optimize Your Events
    • High Availability

What is an Azure Time Series Model?

A Time Series Model is a central entity in Azure Time Series, which lets you manage, maintain, and enhance time series datasets for analysis.

The Time Series Model provides the following capabilities:

  • Create and manage calculations and formulas using scalar functions and aggregate operations
  • Define hierarchical relationships to enable search, cross referencing, and navigation
  • Define properties associated with data instances, and use them to build hierarchies

Time Series Model Components

A Time Series Model has three key components: instances, hierarchies, and types. You use these components to specify a model for analysis of time series data, and organize the data.

Source: Azure

Instances

A Time Series Model instance is an individual time series. It has a unique identifier, the time series ID, which is typically the ID of the asset or device that generated the data.

Each instance can be described by additional properties, such as type, description, name, hierarchies, and instance fields. Instance fields are collections of information such as hierarchy level, device operator, manufacturers, and more.

As soon as you add an event source in Azure Time Series Insights, the system discovers time series and creates Time Series Models. You can update these models or create new ones using Time Series Model queries.

Hierarchies

The Time Series Model organizes instances by specifying attribute names and hierarchical relationships between them. One instance can map to one hierarchy or multiple hierarchies—see an example below.

Source: Azure

Types

Time Series Model types are associated with a specific instance, and allow you to define variables or expressions used to perform calculations.

Types can have one or more variables. For example, one Time Series Model instance could have temperature sensor type, consisting of average temperature, minimum temperature, and maximum temperature variables.

Azure Time Series Data Storage

Azure Time Series Insights works on datasets stored in your Azure Storage account. Here are some key considerations for storing time series datasets in Azure.

Storage Tiers and Data Availability

Azure Time Series Insights Gen2 partitions and indexes data, to optimize query performance. After indexing the data, you can query data from hot storage (if enabled) and cold storage. 

The amount of data captured and the processing speed of each partition affects availability. You can configure alerts to be notified when data processing is lagging in your environment.

Warm Store

You can access data from hot storage through the Time Series Query API, Time Series Insight TSI Explorer, or the Power BI connector. Hot storage queries are free and have no quota, but you can perform up to 30 concurrent requests.

When warm data store is enabled, it works as follows:

  • Ingests all data—if warm store is enabled, all data flowing into the environment is routed to it, regardless of event timestamps. The streaming pipeline is built for near-real-time streaming and does not support collection of past events.
  • Retention period—calculated based on the date and time the event was indexed in warm storage, not the timestamp of the event.
  • No back-filling—if you enable hot storage in an existing environment that already has data in cold storage, the data will not be repopulated into warm storage.

Cold Store

For events sent to cold storage, Azure Time Series Insight Gen2 keeps up to two copies of each event in the Azure Storage account. Events are stored in chronological order. Over time, Azure Time Series Insights Gen2 repartitions your data to optimize high-performance queries. Data is stored in the Azure storage account indefinitely.

Azure Time Series Best Practices

Monitor Azure Time Series Insights

Azure Time Series can be a critical part of IoT data pipelines. It is important to set up monitoring, to ensure the service is working correctly, identify problems and resolve them. You can use Azure Monitor to perform continuous monitoring of Azure Time Series Insights. 

Focus on metrics such as bytes received from all event sources, bytes successfully processed, and bytes available for processing. Also watch the TSIIngress table, which shows errors that occur in the event ingress pipeline.

Using Warm Store and Cold Store

You can choose to enable a “warm store”, which enables faster response times, and provides a retention period of 7 to 30 days. Note that data that needs to be retained for more than 30 days is served from the “cold store”, and incurs a fee for data access. Interactive analytics on recent data should reside in warm storage, while long-term trends and pattern analysis should reside in cold store.

Configure Time Series IDs and Timestamp Properties

In Azure Time Series, you need to select an ID and three keys for each time series, which you can later use to partition the data. You should also designate a timestamp property when adding event sources for future tracking. Otherwise, the event’s enquing time will be used as its timestamp. Also, notice that timestamp values are case-sensitive and should be formatted to individual event source specifications.

Optimize Your Events

Ensure that events are optimized before sending them to Azure Time Series Insights. It is recommended to denormalize events before ingesting them. You should store metadata in your time series model, and ensure instance fields and events only contain vital information, like time series ID and the timestamp property.

High Availability

Time Series Insights leverages region-level redundancies to high availability. You can perform disaster recovery within Azure using Azure Site Recovery (ASR). Other features include geo-replication and load balancing for failover, data recovery, and backup of on-premises or Azure-based VMs using the Azure Backup service.

To ensure your devices and users have global, cross-region high availability, make sure you enable the correct Azure features. 

Conclusion

This article explained about Azure Time Series Insights components, storage, and best practices that can help you derive insights from IoT time series data in the cloud.

FAQ

What is Azure Time Series Insights?

Azure Time Series Gen2 is an end-to-end Internet of Things (IoT) analytics service designed for industrial IoT deployments. It provides powerful APIs you can use to integrate it with current workflows and applications. 

What is an Azure Time Series Model?

A Time Series Model is a central entity in Azure Time Series, which lets you manage, maintain, and enhance time series datasets for analysis.

What are the Time Series Model Components?

– Instances
– Hierarchies
– Types


Real Time Analytics and Stream Processing: An Introduction

Real Time Analytics and Stream Processing: An Introduction
real time analytics

What is Real Time Analytics?

Real-time analytics makes it possible for organizations to capture live streams of data, process them very quickly, and extract insights or perform operations on the data in real time or near-real-time. It is based on stream processing technology that can handle a very high throughput of event data. 

There are two types of real time analytics:

  • On-demand analytics—providing data or computation results to users or applications in real time. For example, displaying the current product price to a user on an eCommerce site.
  • Continuous analytics—processing events on a continuous basis and streaming the results to end users, applications, or a data store. For example, showing live stock market data on a dashboard in a financial institution. 

Real-time analytics has many uses in the digital economy. For example, it can help businesses track customer data and respond with personalized offers, improving customer engagement. It  can enable rapid, automated response to shifts in the market, enabling dynamic pricing on eCommerce sites. Another common use is to process massive volumes of log or sensor data, from IT systems or internet of things (IoT) devices, and using them to drive business decisions.

In this article, you will learn:

  • What are Stream Processing Frameworks?
  • Top Stream Processing Frameworks
    • Amazon Kinesis
    • Azure Stream Analytics
    • Apache Spark
  • Best Practices for Real Time Analytics

Google Firebase Pricing for Dummies

Google Firebase Pricing for Dummies

How Much Does Google Firebase Cost?

Firebase is a platform for building web and mobile applications. It was originally developed by Firebase Inc., and purchased by Google in 2018.

Firebase has two main pricing plans. The Firebase pricing plans are:

  • Spark Plan (Firebase free tier) is a basic plan offering 1 GB total storage, 20K writes/day, 50K reads/day, and 20K deleted/day
  • Blaze Plan (paid) is a pay as you go plan charging $0.18/GB for database storage, $0.026/GB for application storage, and additional charges for database operations, data transfer, etc.

In this article, you will learn:

  • How Much Does Google Firebase Cost?
  • Firebase Always-Free Services
  • Firebase Pricing for Paid Services
  • Cloud Firestore Database
  • Real-Time Databases
  • Hosting
  • Authentication
  • Cloud Functions
  • Firebase ML
  • Test Lab
  • Firebase: Estimating Your Costs

Firebase Database: Should You Choose Realtime DB or Cloud Firestore?

Firebase is a NoSQL cloud service, offered as part of the Google Cloud Platform database offerings. It is based on a document-model and can be used to store and sync data in real-time with horizontal scaling. You can use Firebase for multi-user applications, such as mobile apps, serverless applications, and offline applications. 

Beyond standard NoSQL functionality, Firebase includes features for authentication, crash reporting, messaging, performance monitoring, and analytics. Firebase is accessible through SDK and API, with support for both Unity and C++. You can integrate it with a wide variety of services and utilities, including Kubernetes deployments, BigQuery, Google Marketing Platform, Data Studio, JIRA, and Slack. 

This article will explain the differences between Firestore vs Real Time Database.


Firebase Storage: What It Is and How It Works

Firebase is a web and mobile application development platform powered by Google. Cloud Storage is incorporated natively into the Firebase architecture.

When creating Firebase projects, you’re also creating Google Cloud projects, because this is where Firebase stores data. This integration enables you to leverage Google Cloud security features for Firebase apps, as well as manage media directly through your storage account. 

This article explains how Firebase architecture works, and walks you through the process of  accessing and uploading files, setting up monitoring, and configuring user-based security rules.

In this article, you will learn:

  • What is Firebase storage?
  • Firebase architecture in the cloud
  • Firebase storage methods

5 Ways to Reduce Lambda Cold Starts

AWS Lambda is a serverless service that you can use to run functions on the AWS cloud. Lambda functions can be triggered automatically based on pre-configured events, or manually. Whichever way you choose to run your functions, you need to carefully monitor performance, to ensure that you are not billed for cold starts. Lambda cold starts typically occur when the system cannot meet the demands needed to deploy your function. This article explains the main factors that contribute to cold start times, and offers three techniques you can use to reduce the impact of Lambda cold starts.