Troubleshoot Azure Operator Nexus Bare Metal Machine server problems

This article describes how to troubleshoot server problems by using Restart, Reimage, and Replace actions on Azure Operator Nexus Bare Metal Machines (BMMs). You might need to take these actions on your server for maintenance reasons, which might cause a brief disruption to specific BMMs.

The time required to complete each of these actions is similar. Restarting is the fastest, whereas replacing takes slightly longer. All three actions are simple and efficient methods for troubleshooting.

Prerequisites

• Familiarize yourself with BMM capabilities: Review Bare Metal Machine Platform Commands for comprehensive details on each action, command parameters, and platform behavior. This troubleshooting guide focuses on workflows and scenarios; refer to the platform commands article for complete command reference.
• Gather the following information (necessary for all actions):
  • Name of the managed resource group for the BMM
  • Name of the BMM that requires a lifecycle management operation
  • Subscription ID
• Cluster Detailed status must be Running
• Cluster to Cluster Manager connectivity must be Connected

Identify the corrective action

When troubleshooting a BMM for failures and determining the most appropriate corrective action, it's essential to understand the available options. This article provides a systematic approach to troubleshoot Azure Operator Nexus server problems using these three methods:

• Restart - Least invasive method, best for temporary glitches or unresponsive Virtual Machines (VMs). Preserves OS and data.
• Reimage - Intermediate solution, restores OS to known-good state without hardware changes. Wipes OS disk but tenant data preserved.
• Replace - Most comprehensive action, required after hardware component repairs (RAM, disk, NIC, etc.). Includes hardware validation before provisioning. Choose storage policy based on your scenario:
  • Use --storage-policy DiscardAll for new deployments, motherboard replacements, BMMs offline 30+ days, or BMMs with no workloads
  • Use --storage-policy Preserve to retain tenant data when none of the above conditions apply

Troubleshooting decision tree

Follow this escalation path when troubleshooting BMM issues. Always start with the least disruptive action and escalate only if the problem persists:

Key principles:

• Always validate that the issue is resolved after each corrective action
• Use pre-checks to identify VMs and NAKS nodes on the BMM before starting
• Use post-checks to verify BMM, VM, and NAKS node status after completion
• Consult Hardware Component Replacement Guide to determine if replace is required

Troubleshoot with a restart action

Restarting a BMM is a process of restarting the server through a simple API call. This action can be useful for troubleshooting problems when Tenant VMs on the host aren't responsive or are otherwise stuck.

The restart typically is the starting point for mitigating a problem.

Restart workflow

Follow these steps to safely restart a BMM:

1. Assess impact - Determine if restarting the BMM impacts critical virtual machines or NAKS nodes
2. Choose restart method:
   • If BMM is running: Use restart command (coordinates shutdown and startup as single operation)
   • If BMM is powered off: Use start command
   • If you need manual control: Use power-off followed by start
3. Execute restart - Run the appropriate command
4. Verify status - Check BMM is back online and functioning properly using post-check commands

Example commands (see BMM Platform Commands for complete parameter details):

# Power off (if needed)
az networkcloud baremetalmachine power-off --name <bareMetalMachineName> --resource-group "<resourceGroup>" --subscription <subscriptionID>

# Start a powered-off BMM
az networkcloud baremetalmachine start --name <bareMetalMachineName> --resource-group "<resourceGroup>" --subscription <subscriptionID>

# Restart a running BMM
az networkcloud baremetalmachine restart --name <bareMetalMachineName> --resource-group "<resourceGroup>" --subscription <subscriptionID>

Infrastructure Post Check

To verify the BMM status after restart:

az networkcloud baremetalmachine show \
  --name <bareMetalMachineName>  \
  --resource-group "<resourceGroup>" \
  --subscription <subscriptionID> \
  --query "{name:name, provisioningState:provisioningState, powerState:powerState, readyState:readyState}" -o table

Expected result:

• provisioningState: Succeeded
• powerState: On
• readyState: True

If the BMM doesn't reach this state within the expected timeout (40 minutes), investigate using status and activity log information.

Troubleshoot with a reimage action

Reimaging a BMM is a process that you use to redeploy the image on the OS disk, without affecting the tenant data. This action executes the steps to rejoin the cluster with the same identifiers.

The reimage action can be useful for troubleshooting problems by restoring the OS to a known-good working state. Common causes that can be resolved through reimaging include recovery due to doubt of host integrity, suspected or confirmed security compromise, or "break glass" write activity.

A reimage action is the best practice for lowest operational risk to ensure the integrity of the BMM.

Reimage workflow

Follow these steps to safely reimage a BMM:

1. Verify running virtual machines and NAKS Nodes - Identify what workloads are on the BMM using pre-check commands
2. Assess impact - Determine if VMs/NAKS nodes can tolerate downtime (running VMs will auto-restart after reimage)
3. Cordon and evacuate - Run cordon --evacuate "True" to gracefully drain workloads
4. Perform reimage - Execute the reimage operation (wipes OS disk, preserves tenant data)
5. Monitor progress - Reimage includes deprovisioning, provisioning, and cloud init phases (up to 3 hours)
6. Uncordon - Make the BMM schedulable again after reimage completes
7. Verify status - Use post-check commands to confirm BMM, VMs, and NAKS nodes are healthy

Infrastructure Pre Check

Step 1: Identify VMs and NAKS clusters on the BMM

az networkcloud baremetalmachine show -n <nodeName> \
  --resource-group <resourceGroup> \
  --subscription <subscriptionId> \
  --query "associatedResourceIds" -o json

Example output:

[
  "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroup>/providers/Microsoft.NetworkCloud/virtualMachines/firewallvnf",
  "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroup>/providers/Microsoft.NetworkCloud/kubernetesClusters/naks-01"
]

Understanding the output:

• An empty array [] means no VMs or NAKS clusters are running on this BMM. You can proceed to the cordon step.
• If the array contains VM ARM resource IDs (.../virtualMachines/...), check the status of each VM before proceeding.
• If the array contains NAKS ARM resource IDs (.../kubernetesClusters/...), check the status of each NAKS cluster before proceeding.

Record the ARM IDs returned here so you can use the same IDs during the post-check.

Step 2a: Review VMs on the BMM

Use the ARM ID from the previous output (for resources containing /virtualMachines/):

az networkcloud virtualmachine show \
  --ids "<VM_ARM_ID>" \
  --query "{name:name, detailedStatus:detailedStatus, powerState:powerState}" -o table

Example output (your VM may show Running, Stopped, Provisioning, or Error):

Name         DetailedStatus    PowerState
-----------  ----------------  ------------
firewallvnf  Running           On

Step 2b: Review NAKS nodes on the BMM

Use the ARM ID from the previous output (for resources containing /kubernetesClusters/) to capture basic cluster context:

az networkcloud kubernetescluster show \
  --ids "<NAKS_ARM_ID>" \
  --query "{name:name, detailedStatus:detailedStatus}" -o table

To see which NAKS nodes are running on each BMM, run the following command against your NAKS cluster:

kubectl get nodes -o custom-columns="NODE:.metadata.name,BMM:.metadata.labels.topology\.kubernetes\.io/baremetalmachine"

Example output:

NODE                                    BMM
naks-01-agentpool1-md-tjr8k-4qqx4        rack1-compute01
naks-01-agentpool2-md-zzrw8-gg8rx        rack1-compute01
naks-01-agentpool2-md-zzrw8-lqrmt        rack1-compute02
naks-01-control-plane-rmwqr              rack1-compute02

Review the output and identify nodes where the BMM column matches the BMM you plan to reimage. In this example, if you're reimaging rack1-compute01, two nodes will be affected: naks-01-agentpool1-md-tjr8k-4qqx4 and naks-01-agentpool2-md-zzrw8-gg8rx.

Reimage

The following Azure CLI command will cordon the specified bareMetalMachineName.

az networkcloud baremetalmachine cordon \
  --evacuate "True" \
  --name <bareMetalMachineName> \
  --resource-group "<resourceGroup>" \
  --subscription <subscriptionID>

The following Azure CLI command will reimage the specified bareMetalMachineName.

az networkcloud baremetalmachine reimage \
  --name <bareMetalMachineName>  \
  --resource-group "<resourceGroup>" \
  --subscription <subscriptionID>

The following Azure CLI command will uncordon the specified bareMetalMachineName.

az networkcloud baremetalmachine uncordon \
  --name <bareMetalMachineName> \
  --resource-group "<resourceGroup>" \
  --subscription <subscriptionID>

Infrastructure Post Check

Step 1: Verify BMM status after reimage:

az networkcloud baremetalmachine show \
  --name <bareMetalMachineName>  \
  --resource-group "<resourceGroup>" \
  --subscription <subscriptionID> \
  --query "{name:name, provisioningState:provisioningState, powerState:powerState, readyState:readyState}" -o table

A provisioningState of Succeeded and readyState of True indicates the BMM has been provisioned and rejoined the cluster.

Step 2: Verify VM status

Check that VMs previously running on this BMM have restarted successfully:

az networkcloud virtualmachine show \
  --ids "<VM_ARM_ID>" \
  --query "{name:name, detailedStatus:detailedStatus, powerState:powerState}" -o table

Step 3: Verify NAKS node status

Validate node-level recovery (for example, that nodes are Ready and running on the expected BMMs) by running the following command against your NAKS cluster:

kubectl get nodes -o custom-columns="NODE:.metadata.name,READY:.status.conditions[?(@.type=='Ready')].status,BMM:.metadata.labels.topology\.kubernetes\.io/baremetalmachine"

Troubleshoot with a Replace action

Servers contain many physical components that can fail over time. It's important to understand which physical repairs require BMM replacement and when BMM replacement is recommended. The impact on tenant data depends on the --storage-policy value you choose — see the Storage policy decision guide below to determine the correct value for your scenario.

A hardware validation process is invoked to ensure the integrity of the physical host in advance of deploying the OS image.

Related articles:

• Hardware Validation Overview - Overview of the hardware validation process
• Troubleshoot Hardware Validation - Check and troubleshoot hardware validation results

Replace workflow

Before committing to a Replace action, it's important to diagnose the root cause of the failure. In many cases, a Troubleshooting Report (TSR) should be collected and reviewed first to confirm that Replace is the correct corrective action and to identify any mitigations that should be applied before or during the replace operation.

Collect a Troubleshooting Report (TSR)

A TSR (also known as a Dell SupportAssist collection) gathers detailed hardware diagnostic data from the BMM, including system information, hardware inventory, lifecycle controller logs, and storage controller status. Reviewing a TSR before performing a Replace helps you:

• Confirm the root cause of a provisioning or hardware failure
• Identify which components need physical repair or replacement
• Determine if Replace is necessary or if a less disruptive action (restart, reimage) would resolve the issue
• Inform your storage policy choice by understanding the state of disks and RAID controllers

To collect a TSR, use the hardware-support-data-collection command as described in SupportAssist or TSR collection for Dell troubleshooting.

After reviewing the TSR findings and completing any physical repairs or mitigations, proceed with the replace workflow below.

Replace workflow steps

Follow these steps to safely replace a BMM after hardware repairs:

1. Identify hardware replaced - Determine which components were repaired (see Hardware Component Replacement Guide)
2. Assess data impact - If storage components were replaced (SSD/PERC/System board/Backplane), VM data may be lost; migrate VMs before replace
3. Choose storage policy - Select the correct --storage-policy value based on your scenario (see Storage policy decision guide below)
4. Check BMM health - Determine if BMM is healthy or unresponsive using pre-check commands
5. Cordon and evacuate - If BMM is unhealthy or unresponsive, skip to step 7. If BMM is healthy, run cordon --evacuate "True" to drain workloads, or use --safeguard-mode None in step 7 which handles evacuation automatically
6. Verify firmware - Ensure all replaced components meet minimum firmware requirements
7. Execute replace - Run replace command with required parameters (includes hardware validation phase)
8. Monitor progress - Replace includes hardware validation, deprovisioning, provisioning, and cloud init phases (up to 4 hours)
9. Uncordon - Make the BMM schedulable again after replacement completes
10. Verify status - Use post-check commands to confirm BMM, VMs, and NAKS nodes are healthy

Storage policy decision guide

Choosing the correct --storage-policy value is critical. The wrong choice can result in unnecessary data loss or a failed replace operation. Use the following table to determine which value to use. For the full reference, see Choose the right storage policy.

Example cordon command (see Cordon for complete details):

az networkcloud baremetalmachine cordon --evacuate "True" --name <bareMetalMachineName> --resource-group "<resourceGroup>" --subscription <subscriptionID>

Infrastructure Pre Check

Step 1: Check BMM health status

az networkcloud baremetalmachine show -n <nodeName> \
  --resource-group <resourceGroup> \
  --subscription <subscriptionId> \
  --query "{name:name, powerState:powerState, readyState:readyState, detailedStatus:detailedStatus}" -o table

Step 2: Identify VMs and NAKS clusters on the BMM

az networkcloud baremetalmachine show -n <nodeName> \
  --resource-group <resourceGroup> \
  --subscription <subscriptionId> \
  --query "associatedResourceIds" -o json

Step 3: Assess VM and NAKS impact based on hardware replaced

Step 4: Optional verification (healthy BMM only)

Use the checks below to record a baseline of what is running on the BMM so you can compare results after replace.

If replacing storage components (SSD / PERC / System board / Backplane): VM data may be permanently lost. Delete affected VMs and recreate them on a healthy BMM before running replace. You can skip the VM status check for VMs you plan to recreate.

For VMs:

az networkcloud virtualmachine show \
  --ids "<VM_ARM_ID>" \
  --query "{name:name, detailedStatus:detailedStatus, powerState:powerState}" -o table

For NAKS clusters:

az networkcloud kubernetescluster show \
  --ids "<NAKS_ARM_ID>" \
  --query "{name:name, detailedStatus:detailedStatus}" -o table

Refer to the Reimage Pre Check section for detailed guidance on interpreting VM and NAKS status values.

Hardware component replacement guide

When you're performing a physical hot swappable power supply repair, a replace action isn't required because the BMM host will continue to function normally after the repair.

When you're performing the following physical repairs, we recommend a replace action, though it isn't necessary to bring the BMM back into service:

• CPU
• Dual In-Line Memory Module (DIMM)
• Fan
• Expansion board riser
• Transceiver
• Ethernet or fiber cable replacement

When you're performing the following physical repairs, a replace action is required to bring the BMM back into service:

• Backplane
• System board
• SSD disk
• PERC/RAID adapter
• Mellanox Network Interface Card (NIC)
• Broadcom embedded NIC

After physical repairs are completed, verify that all firmware versions align with the versions supported for the corresponding runtime. Apply the same best-practice checks used during deployment to ensure the components meet the minimum firmware requirements.

Once the firmware checks are complete, proceed with the replace action.

Example replace command (see Replace a Bare Metal Machine for all parameters and safeguard-mode details):

# Choose storage-policy based on your scenario:
#   DiscardAll — new deployment, motherboard replaced, offline 30+ days, or no workloads
#   Preserve  — existing instance with workloads and none of the above conditions apply
# See the storage policy decision guide above for full guidance.

az networkcloud baremetalmachine replace \
  --name <bareMetalMachineName> \
  --resource-group "<resourceGroup>" \
  --bmc-credentials password=<IDRAC_PASSWORD> username=<IDRAC_USER> \
  --bmc-mac-address <IDRAC_MAC> \
  --boot-mac-address <PXE_MAC> \
  --machine-name <OS_HOSTNAME> \
  --serial-number <SERIAL_NUM> \
  --subscription <subscriptionID> \
  --storage-policy <"Preserve" or "DiscardAll">

# Uncordon to allow scheduling
az networkcloud baremetalmachine uncordon --name <bareMetalMachineName> --resource-group "<resourceGroup>" --subscription <subscriptionID>

Infrastructure Post Check

Step 1: Verify BMM status after replace:

az networkcloud baremetalmachine show \
  --name <bareMetalMachineName>  \
  --resource-group "<resourceGroup>" \
  --subscription <subscriptionID> \
  --query "{name:name, provisioningState:provisioningState, powerState:powerState, readyState:readyState}" -o table

A provisioningState of Succeeded and readyState of True indicates the BMM has been provisioned and rejoined the cluster.

Step 2: Verify VM status:

Check that VMs previously running on this BMM have restarted successfully:

az networkcloud virtualmachine show \
  --ids "<VM_ARM_ID>" \
  --query "{name:name, detailedStatus:detailedStatus, powerState:powerState}" -o table

Step 3: Verify NAKS node status

Validate node-level recovery (for example, that nodes are Ready and running on the expected BMMs) by running the following command against your NAKS cluster:

kubectl get nodes -o custom-columns="NODE:.metadata.name,READY:.status.conditions[?(@.type=='Ready')].status,BMM:.metadata.labels.topology\.kubernetes\.io/baremetalmachine"

Summary

Restarting, reimaging, and replacing are effective troubleshooting methods for addressing Azure Operator Nexus server problems. Here's a quick reference guide:

Best practices

• Always follow the escalation path: Start with restart, then reimage, then replace unless the issue clearly indicates otherwise.
• Verify virtual machines and NAKS nodes before action: Use the provided commands to identify running virtual machines and NAKS nodes before any disruptive action.
• Cordon with evacuation: When performing reimage or replace actions, always use cordon with evacuate="True".
• Choose the correct storage policy for replace: Use the storage policy decision guide to determine whether to use DiscardAll or Preserve. Choosing the wrong value can result in unnecessary data loss or a failed replace.
• Never run multiple operations simultaneously: Ensure one operation completes before starting another to prevent server issues.
• Verify resolution: After performing any action, verify the BMM status and that the original issue is resolved.

For comprehensive details on all BMM actions, including command parameters, timeout values, and platform behavior, see Bare Metal Machine Platform Commands.

If you still have questions, contact support. For more information about Support plans, see Azure Support plans.