Skip to main content
Version: 1.2.0

Deployment: Via Azure Arc Extension

This guide explains how to deploy DataStream using Azure Arc Extensions, enabling consistent management across hybrid and multi-cloud environments. This deployment model is ideal for organizations with resources spanning on-premises datacenters, multiple cloud providers, or edge environments.

Benefits​

Deploying DataStream as an Azure Arc Extension provides several advantages:

  • Unified management: Manage all DataStream deployments from a single control plane
  • Hybrid consistency: Deploy identical configurations across on-premises and cloud environments
  • Central governance: Apply consistent policies and compliance rules
  • Simplified operations: Automated deployment and updates across diverse infrastructure
  • Azure integration: Leverage Azure's security, monitoring, and management capabilities

Architecture Overview​

Azure Arc extends Azure's control plane to non-Azure environments. The DataStream extension architecture consists of:

  1. Azure Arc-enabled servers: On-premises or other cloud VMs registered with Azure Arc
  2. DataStream extension: The extension package containing DataStream software
  3. Azure Resource Manager: Central control plane for deployment and configuration
  4. Azure Policy: Optional component for automated compliance and configuration

Prerequisites​

Before deploying DataStream via Azure Arc, ensure you have:

  1. Azure Arc-enabled servers: Register target servers with Azure Arc
  2. Required permissions: Contributor role on resource groups
  3. Network connectivity: Outbound access from servers to Azure services
  4. System requirements:
  • OS: Windows Server 2012 R2+ or Linux (Ubuntu 16.04+, RHEL 7+, etc.)
  • CPU: 2+ cores recommended
  • Memory: 4GB+ RAM
  • Storage: 10GB+ free space

Deployment Steps​

1. Register Azure Arc Servers​

If not already done, register servers with Azure Arc:

  1. Install Azure CLI and the Arc extension:

    • Install Azure CLI if not already installed

      Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi
      Start-Process msiexec.exe -Wait -ArgumentList '/I AzureCLI.msi /quiet'

    • Add Arc extension

      az extension add --name connectedmachine

  2. Generate onboarding script from Azure Portal or CLI:

    az connectedmachine generate-installation-script --resource-group myResourceGroup --location eastus --output-dir .

  3. Run the script on each target server:

    Run the generated script

    .\OnboardingScript.ps1

2. Register DataStream Extension​

  1. Register the DataStream extension provider in your Azure subscription:

    az provider register --namespace Microsoft.HybridCompute
    az provider register --namespace Microsoft.GuestConfiguration
    az provider register --namespace Microsoft.Extensions

  2. Create an extension deployment template:

    Create a new ARM template for DataStream extension

    @"
    {
      "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
      "contentVersion": "1.0.0.0",
      "parameters": {
        "machineName": {
          "type": "string",
          "metadata": {
            "description": "Name of the Azure Arc-enabled server"
          }
        },
        "location": {
          "type": "string",
          "metadata": {
            "description": "Location of the machine"
          }
        }
      },
      "resources": [
        {
          "type": "Microsoft.HybridCompute/machines/extensions",
          "name": "[concat(parameters('machineName'), '/DataStreamExtension')]",
          "apiVersion": "2020-08-15-preview",
          "location": "[parameters('location')]",
          "properties": {
            "publisher": "DataStream",
            "type": "CustomExtension",
            "autoUpgradeMinorVersion": true,
            "settings": {
                "devices": [
                {
                  "id": 1,
                  "name": "syslog_collector",
                  "type": "syslog",
                  "properties": {
                  "port": 514
                  }
                }
                ],
              "targets": [
                {
                  "name": "azure_monitor",
                  "type": "azure_monitor",
                  "workspace_id": "${WORKSPACE_ID}",
                  "workspace_key": "${WORKSPACE_KEY}"
                }
              ]
            },
            "protectedSettings": {
              "WORKSPACE_ID": "<your-workspace-id>",
              "WORKSPACE_KEY": "<your-workspace-key>"
            }
          }
        }
      ]
    }
    "@ | Out-File -FilePath "datastream-extension.json" -Encoding utf8

3. Deploy the Extension​

  1. Deploy to a single server using Azure CLI:

    az deployment group create `
      --resource-group myResourceGroup `
      --template-file datastream-extension.json `
      --parameters machineName=myArcServer location=eastus

  2. Deploy to multiple servers using Azure Policy:

    • Navigate to Azure Policy in the portal
    • Create a new policy definition for extension deployment
    • Assign policy to a management group or subscription
    • Target Arc-enabled servers using appropriate filters

4. Configure DataStream Settings​

Configure DataStream through the extension settings:

  1. Use Azure Portal:

    • Navigate to Arc-enabled server > Extensions
    • Select DataStream extension
    • Update configuration parameters

  2. Use Azure Resource Manager template:

    # Create updated configuration template
    @"
    {
      "settings": {
        "collectors": [
          {
            "id": 1,
            "name": "windows_collector",
            "type": "windows",
            "properties": {
              "address": "localhost",
              "username": "datastream",
              "password": "${WINDOWS_PASSWORD}"
            }
          }
        ],
        "pipelines": [
          {
            "name": "windows_events",
            "processors": [
              {
                "grok": {
                  "field": "message",
                  "patterns": ["%{WINLOG}"]
                }
              }
            ]
          }
        ]
      },
      "protectedSettings": {
        "WINDOWS_PASSWORD": "your-secure-password"
      }
    }
    "@ | Out-File -FilePath "updated-config.json" -Encoding utf8

  3. Apply configuration update:

    az deployment group create `
      --resource-group myResourceGroup `
      --template-file updated-config.json

5. Verify Deployment​

  1. Check extension status via Azure Portal:

    • Navigate to Arc-enabled server > Extensions
    • Verify DataStream extension shows "Provisioning succeeded"

  2. Check extension logs on the server:

    # Check log files in the extension directory
    Get-Content -Path "C:\ProgramData\AzureConnectedMachineAgent\Log\DataStreamExtension\*.log"

  3. Verify DataStream service is running:

    Get-Service DataStream

Managing at Scale​

For large-scale deployments across hybrid environments:

1. Configuration Management​

Use Azure Arc's configuration capabilities:

  1. Create configuration profiles:

    • Define standard DataStream configurations for different server types
    • Use parameters for environment-specific values

  2. Implement Azure Policy for automated deployment:

    • Assign DataStream extension policies to management groups
    • Use policy remediation to ensure compliance

  3. Use GitHub integration for GitOps-based configuration:

    • Store configurations in a Git repository
    • Automatically apply changes when configuration is updated

2. Monitoring and Updates​

  1. Centralized monitoring:

    • Deploy Azure Monitor agent extension alongside DataStream
    • Collect logs and metrics for all deployments
    • Create dashboards for cross-environment visibility

  2. Update management:

    • Configure automatic extension updates for security patches
    • Stage updates across environments (dev/test/prod)
    • Use maintenance windows to minimize disruption

3. Security Management​

  1. Credential management:

    • Use protected settings for sensitive values
    • Rotate credentials automatically using Azure Key Vault

  2. Network security:

    • Implement resource firewalls consistently across environments
    • Monitor network traffic with Azure Network Watcher

Edge and Disconnected Scenarios​

For deployments in edge or occasionally connected environments:

  1. Configure offline mode:

    "settings": {
      "offlineMode": true,
      "localBuffering": {
        "enabled": true,
        "maxSize": "5GB",
        "flushInterval": "4h"
      }
    }

  2. Enable data buffering: Configure local storage for data persistence during disconnection

  3. Connection resiliency: Implement auto-retry mechanisms for reconnection

Transitioning from Other Deployment Methods​

If migrating from other deployment models to Azure Arc:

  1. Preserve configurations - Export existing DataStream configurations
  2. Register servers - Onboard servers to Azure Arc without disrupting current services
  3. Stage migrations - Deploy extension alongside existing installation before cutover
  4. Validate - Verify data flow and performance before decommissioning original deployment

This deployment approach is ideal for organizations with hybrid infrastructure models, providing a consistent management experience regardless of where DataStream is running.