SSH Access to Azure VMs: Keys, Security, and Best Practices

Key-based authentication: how it works

SSH key authentication uses a cryptographic key pair: a private key you keep on your local machine and a public key placed on the remote server. When you connect, SSH proves you hold the private key without transmitting it. The server checks that the public key you claim to hold matches one in its ~/.ssh/authorized_keys file and grants access.

Azure Marketplace Linux images are configured to accept SSH key authentication by default and to disable password authentication. This is the correct default — password authentication exposes your VM to brute-force attacks.

Generating and using SSH keys

Generate a new RSA key pair locally:

# Generate a 4096-bit RSA key pair
ssh-keygen -t rsa -b 4096 -C "your-email@example.com" -f ~/.ssh/azure-vm-key

# This creates two files:
# ~/.ssh/azure-vm-key       (private key — never share this)
# ~/.ssh/azure-vm-key.pub   (public key — placed on the server)

When creating a VM, pass the public key:

az vm create \
  --resource-group my-rg \
  --name my-vm \
  --image Ubuntu2204 \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/azure-vm-key.pub

Connect using the private key:

ssh -i ~/.ssh/azure-vm-key azureuser@<vm-public-ip>

If you used —generate-ssh-keys instead, the key was saved to ~/.ssh/id_rsa and SSH uses it automatically without the -i flag. Using a named key per VM gives you better control — you can revoke access to one VM without affecting others.

Using SSH config for multiple VMs

Typing the full IP, username, and key path every time is tedious. The SSH config file at ~/.ssh/config lets you define aliases:

# ~/.ssh/config
Host my-azure-vm
  HostName 20.x.x.x
  User azureuser
  IdentityFile ~/.ssh/azure-vm-key

Host dev-server
  HostName 20.y.y.y
  User azureuser
  IdentityFile ~/.ssh/dev-key
  StrictHostKeyChecking no

With this config, you connect with just ssh my-azure-vm instead of the full command. This is particularly useful when you manage multiple VMs regularly.

Restricting SSH network access

Opening port 22 to 0.0.0.0/0 in a Network Security Group invites automated attacks within minutes of a VM going live. Every internet-accessible SSH port accumulates login attempts continuously. The common defences:

Restrict to your IP

# Update the SSH NSG rule to only allow your current IP
MY_IP=$(curl -s https://api.ipify.org)

az network nsg rule update \
  --resource-group my-rg \
  --nsg-name my-vm-nsg \
  --name default-allow-ssh \
  --source-address-prefixes "$MY_IP"

This works for individual developers but breaks when your IP changes or multiple team members need access from different locations.

Use Azure Bastion (recommended)

Azure Bastion provides browser-based SSH access through the Azure portal, with no public IP needed on the VM and no port 22 open to the internet. See Azure Bastion overview for setup and cost.

Use a VPN or private network

Connect to the Azure VNet over VPN and SSH to the VM’s private IP. This removes public IP from the VM entirely. Good for teams that already use Azure VPN Gateway or ExpressRoute.

SSH agent forwarding for private subnets

If you have a bastion host (a VM with public access used as a jump server) and need to reach VMs in private subnets, SSH agent forwarding lets you authenticate to the private VM using your local key without copying the private key to the bastion host.

# Start the SSH agent and add your key
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/azure-vm-key

# Connect to the bastion with agent forwarding enabled
ssh -A azureuser@<bastion-public-ip>

# From inside the bastion, SSH to the private VM
# (uses your local key via forwarding, not a key on the bastion)
ssh azureuser@10.0.1.5

Managing keys across a team

When multiple engineers need SSH access to VMs, the naive approach is adding each person’s public key to authorized_keys manually. This becomes unmaintainable: when someone leaves the team, you must find and remove their key from every VM.

Better approaches:

• Azure AD SSH login. Install the AADSSHLoginForLinux VM extension and use az ssh vm to authenticate with Azure AD credentials. Access is controlled through Azure RBAC rather than SSH key files — add or revoke access by changing role assignments, not by touching key files on VMs.
• Use a jump host with RBAC-managed access. Only the jump host needs individual SSH keys. Private VMs are accessed via the jump host’s service account. Team members get access to the jump host, not to individual VMs.
• Use Azure Bastion in native client mode. Supports certificate-based auth with Entra ID for teams, centralised access logging, and no long-lived SSH keys on VMs.