Azure industrial IoT data analysis

This article discusses data analysis and visualizations in an Azure industrial IoT (IIoT) analytics solution. There are several ways to analyze, query, and present industrial data by using visualizations and dashboards. You can use these tools to evaluate solution effectiveness, explore trends, and derive insights. The insights you gain can power advanced queries, machine learning model training, or logic apps that take actions.

Your IIoT analytics solution might use some or all of the following options, depending on what you need to do with your data.

• Ad-hoc analytics and trend visualizations with Azure Data Explorer dashboards (Preview).
• Power BI visualizations and dashboards. Connect to your Azure IIoT data by using the Azure Data Explorer connector for Power BI. Use Power BI to combine external data from your Enterprise Resource Planning (ERP), Enterprise Asset Management (EAM), or other line-of-business systems with your IIoT data.
• Custom web applications, for advanced visualizations like schematic views and process graphics.
• Microsoft and GitHub notebooks that work with open-source tools like Jupyter Notebook and Matplotlib.

Architecture

The following diagram shows how analytics and visualization tools collect, process, store, and analyze warm, cold, and historical data sources.

1. Devices send telemetry data to the cloud through Azure IoT Hub.
2. IoT Hub sends the device telemetry to a data processing engine, and time series metadata to the time series model store.
3. The data processing engine routes data into warm storage and cold storage. The engine sends time series IDs to the time series model store.
4. The query API can query over the warm storage, cold storage, and time series data.
5. Query results feed into data explorer dashboards, visualizations, and third-party apps.
6. Advanced analytics tools and machine learning also use cold storage data.
7. Telemetry metadata and query results continually update the time series model.

Azure Data Explorer

Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Azure Data Explorer is ideally suited to explore, analyze, and visualize raw data from industrial systems.

Azure Data Explorer provides a web application, the Web UI, where you can run queries and build dashboards. For more information, see Visualize data with Azure Data Explorer dashboards (Preview). Azure Data Explorer also integrates with other dashboard services like Power BI.

Azure Data Explorer can ingest data from Azure IoT Hub, Azure Event Hubs, Azure Stream Analytics, Power Automate, Azure Logic Apps, Kafka, Apache Spark, and many other services and platforms. Ingestion is scalable, and there are no limits.

Supported Azure Data Explorer ingestion formats include JSON, CSV, Avro, Parquet, ORC, TXT, and other formats. For more information, see Data formats supported by Azure Data Explorer for ingestion.

Azure Data Explorer supports:

• Optimized hot storage on compute nodes.
• Cold storage in the subscription's Azure Blob Storage account.
• Automatic continuous data export to Azure Storage.
• Distributed columnar storage and retention.
• External tables to query exported data.
• Data querying in KQL and SQL.
• Data visualization formats including Azure Data Explorer Dashboards, Power BI, Grafana, and other visualization tools that use ODBC and JDBC connectors.

The optimized native Azure Data Explorer connector for Power BI supports direct query or import mode, including query parameters and filters.

For machine learning (ML), Azure Data Explorer supports R or Python to export ML models for building new models or scoring data. Azure Data Explorer has native capabilities for forecasting, anomaly detection, and clustering for diagnostics and root cause analysis (RCA).

For security, Azure Data Explorer supports virtual network injection, Private Link, and encryption at rest with customer-managed keys. Azure Data Explorer includes granular role-based access control (RBAC) roles for functions and data access, row-level security (RLS), and data masking. Azure Data Explorer is built on Azure Blob Storage for Azure-supported 99.9% availability.

Power BI

Power BI is an ideal visualization solution for dashboards that show factory key progress indicators (KPIs). You can use the Azure Data Explorer connector for Power BI to connect to industrial data stored in Azure Data Explorer. Power BI provides powerful reporting and dashboard capabilities that let you share insights and results across your organization. Power BI has desktop, web, and mobile interfaces.

By connecting your data to Power BI, you can:

• Perform correlations with other data sources that Power BI supports, and use many different data visualization options.
• Create Power BI dashboards and reports that use your Azure Data Explorer data, and share them with your organization.
• Unlock data interoperability scenarios simply and easily, with features like suggested Q&A and automatic insights.
• Interact with Azure Data Explorer data by using the powerful Advanced Query Editor in Power BI.

Custom web application

For advanced visualizations, such as schematic views or process graphics, you can create a custom web application. A custom web application can give you a single pane of glass (SPOG) user experience and other advanced capabilities. You can create applications such as:

• Simplified and integrated authoring experiences for Stream Analytics jobs and Azure Logic Apps.
• Process or custom visuals that display real-time data.
• Web apps with embedded Power BI dashboards that display KPIs and external data.
• Visual alert displays using SignalR.
• Administrative applications for adding or removing solution users.

You can create a single-page application (SPA) by using:

Notebooks

Jupyter Notebook is an open-source web application for creating and sharing notebooks. These documents can contain live code, equations, visualizations, persistent data, and narrative text. Jupyter Notebook supports data sources including Azure Data Explorer, Azure Monitor logs, and Application Insights.

For more information, see:

Next steps