Quickstart: Connect an existing Kubernetes cluster to Azure Arc

• An Azure account with an active subscription. Create an account for free.

• A basic understanding of Kubernetes core concepts.

• An identity (user or service principal) that you can use to sign in to Azure CLI and connect your cluster to Azure Arc.

• The latest version of Azure CLI.

• The latest version of connectedk8s Azure CLI extension, installed by running the following command:

  az extension add --name connectedk8s
  

• A running Kubernetes cluster. If you don't have one, you can create a cluster by using one of these options:

  • Kubernetes in Docker (KIND)

  • Use Docker for Linux, Mac, or Windows

  • Self-managed Kubernetes cluster using Cluster API

    Note

    The cluster needs to have at least one node of operating system and architecture type linux/amd64 and/or linux/arm64. See Cluster requirements for more about ARM64 scenarios.

• At least 850 MB free for the Arc agents that you deploy on the cluster, and capacity to use approximately 7% of a single CPU.

• A kubeconfig file and context pointing to your cluster. For more information, see Configure access to multiple clusters.

• An Azure account with an active subscription. Create an account for free.

• A basic understanding of Kubernetes core concepts.

• An identity (user or service principal) that you can use to sign in to Azure PowerShell and connect your cluster to Azure Arc.

• Azure PowerShell version 6.6.0 or later

• The Az.ConnectedKubernetes Azure PowerShell module, installed by running the following command:

  Install-Module -Name Az.ConnectedKubernetes
  

• A running Kubernetes cluster. If you don't have one, you can create a cluster by using one of these options:

  • Kubernetes in Docker (KIND)

  • Use Docker for Linux, Mac, or Windows

  • Self-managed Kubernetes cluster using Cluster API

    Note

    The cluster needs to have at least one node of operating system and architecture type linux/amd64 and/or linux/arm64. See Cluster requirements for more about ARM64 scenarios.

• At least 850 MB free for the Arc agents that you deploy on the cluster, and capacity to use approximately 7% of a single CPU.

• A kubeconfig file and context pointing to your cluster. For more information, see Configure access to multiple clusters.

1. Enter the following commands:

   az provider register --namespace Microsoft.Kubernetes
   az provider register --namespace Microsoft.KubernetesConfiguration
   az provider register --namespace Microsoft.ExtendedLocation
   

2. Monitor the registration process. Registration can take up to 10 minutes.

   az provider show -n Microsoft.Kubernetes -o table
   az provider show -n Microsoft.KubernetesConfiguration -o table
   az provider show -n Microsoft.ExtendedLocation -o table
   

   When the registration finishes, the RegistrationState for these namespaces changes to Registered.

1. Enter the following commands:

   Register-AzResourceProvider -ProviderNamespace Microsoft.Kubernetes
   Register-AzResourceProvider -ProviderNamespace Microsoft.KubernetesConfiguration
   Register-AzResourceProvider -ProviderNamespace Microsoft.ExtendedLocation
   

2. Monitor the registration process. Registration can take up to 10 minutes.

   Get-AzResourceProvider -ProviderNamespace Microsoft.Kubernetes
   Get-AzResourceProvider -ProviderNamespace Microsoft.KubernetesConfiguration
   Get-AzResourceProvider -ProviderNamespace Microsoft.ExtendedLocation
   

   When the registration finishes, the RegistrationState for these namespaces changes to Registered.

az group create --name AzureArcTest --location EastUS --output table

Output:

Location    Name
----------  ------------
eastus      AzureArcTest

New-AzResourceGroup -Name AzureArcTest -Location EastUS

Output:

ResourceGroupName : AzureArcTest
Location          : eastus
ProvisioningState : Succeeded
Tags              :
ResourceId        : /subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/AzureArcTest

az connectedk8s connect --name AzureArcTest1 --resource-group AzureArcTest

Output:

Helm release deployment succeeded

    {
      "aadProfile": {
        "clientAppId": "",
        "serverAppId": "",
        "tenantId": ""
      },
      "agentPublicKeyCertificate": "xxxxxxxxxxxxxxxxxxx",
      "agentVersion": null,
      "connectivityStatus": "Connecting",
      "distribution": "gke",
      "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/AzureArcTest/providers/Microsoft.Kubernetes/connectedClusters/AzureArcTest1",
      "identity": {
        "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "type": "SystemAssigned"
      },
      "infrastructure": "gcp",
      "kubernetesVersion": null,
      "lastConnectivityTime": null,
      "location": "eastus",
      "managedIdentityCertificateExpirationTime": null,
      "name": "AzureArcTest1",
      "offering": null,
      "provisioningState": "Succeeded",
      "resourceGroup": "AzureArcTest",
      "tags": {},
      "totalCoreCount": null,
      "totalNodeCount": null,
      "type": "Microsoft.Kubernetes/connectedClusters"
    }

New-AzConnectedKubernetes -ClusterName AzureArcTest1 -ResourceGroupName AzureArcTest -Location eastus

Output:

Location Name          Type
-------- ----          ----
eastus   AzureArcTest1 microsoft.kubernetes/connectedclusters

1. On the deployment machine, set the environment variables needed for Azure CLI to use the outbound proxy server:

   export HTTP_PROXY=<proxy-server-ip-address>:<port>
   export HTTPS_PROXY=<proxy-server-ip-address>:<port>
   export NO_PROXY=<cluster-apiserver-ip-address>:<port>
   

2. On the Kubernetes cluster, run the connect command with the --proxy-https and --proxy-http parameters specified. If your proxy server is set up with both HTTP and HTTPS, be sure to use --proxy-http for the HTTP proxy and --proxy-https for the HTTPS proxy. If your proxy server only uses HTTP, you can use that value for both parameters.

   az connectedk8s connect --name <cluster-name> --resource-group <resource-group> --proxy-https https://<proxy-server-ip-address>:<port> --proxy-http http://<proxy-server-ip-address>:<port> --proxy-skip-range <excludedIP>,<excludedCIDR> --proxy-cert <path-to-cert-file>
   

   Note

   • Some network requests, such as the ones involving in-cluster service-to-service communication, need to be separated from the traffic that is routed via the proxy server for outbound communication. Use the --proxy-skip-range parameter to specify the CIDR range and endpoints in a comma-separated way so that any communication from the agents to these endpoints doesn't go via the outbound proxy. At a minimum, specify the CIDR range of the services in the cluster as the value for this parameter. For example, let's say kubectl get svc -A returns a list of services where all the services have ClusterIP values in the range 10.0.0.0/16. Then the value to specify for --proxy-skip-range is 10.0.0.0/16,kubernetes.default.svc,.svc.cluster.local,.svc.
   • Most outbound proxy environments expect --proxy-http, --proxy-https, and --proxy-skip-range. The --proxy-cert parameter is only required if you need to inject trusted certificates that the proxy expects into the trusted certificate store of agent pods.
   • The outbound proxy must be configured to allow websocket connections.

   For outbound proxy servers, if you're only providing a trusted certificate, run az connectedk8s connect with just the --proxy-cert parameter specified:

   az connectedk8s connect --name <cluster-name> --resource-group <resource-group> --proxy-cert <path-to-cert-file>
   

   If there are multiple trusted certificates, combine the certificate chain (Leaf cert, Intermediate cert, Root cert) into a single file which you pass in the --proxy-cert parameter.

   Tip

   • --custom-ca-cert is an alias for --proxy-cert. Use either parameter interchangeably. If you pass both parameters in the same command, the one passed last is honored.

1. On the deployment machine, set the environment variables needed for Azure PowerShell to use the outbound proxy server:

   $Env:HTTP_PROXY = "<proxy-server-ip-address>:<port>"
   $Env:HTTPS_PROXY = "<proxy-server-ip-address>:<port>"
   $Env:NO_PROXY = "<cluster-apiserver-ip-address>:<port>"
   

2. On the Kubernetes cluster, run the connect command with the proxy parameter specified:

   New-AzConnectedKubernetes -ClusterName <cluster-name> -ResourceGroupName <resource-group> -Location eastus -Proxy 'https://<proxy-server-ip-address>:<port>'
   

Azure PowerShell doesn't currently support passing in only the proxy certificate without proxy server endpoint details.

az connectedk8s list --resource-group AzureArcTest --output table

Output:

Name           Location    ResourceGroup
-------------  ----------  ---------------
AzureArcTest1  eastus      AzureArcTest

Get-AzConnectedKubernetes -ResourceGroupName AzureArcTest

Output:

Location Name          Type
-------- ----          ----
eastus   AzureArcTest1 microsoft.kubernetes/connectedclusters

To delete the Azure Arc-enabled Kubernetes resource, along with any associated configuration resources and agents running on the cluster, use the following command:

az connectedk8s delete --name AzureArcTest1 --resource-group AzureArcTest

If the deletion process fails, use the following command to force deletion. Add -y to bypass the confirmation prompt:

az connectedk8s delete -n AzureArcTest1 -g AzureArcTest --force

Use this command if you experience problems when creating a new cluster deployment due to previously created resources that aren't completely removed.

You can delete the Azure Arc-enabled Kubernetes resource, any associated configuration resources, and any agents running on the cluster by using the following command:

Remove-AzConnectedKubernetes -ClusterName AzureArcTest1 -ResourceGroupName AzureArcTest