Automating SSH Host Key Rotation in Hybrid Cloud Environments

Introduction

SSH host keys are the first line of trust when a client connects to a server. If a key is compromised or simply ages, continuing to use it defeats the purpose of SSH hardening. In hybrid environments the challenge multiplies: you must rotate keys on bare‑metal racks, private‑cloud VMs, and public‑cloud instances without breaking existing automation or user workflows.

This guide shows you how to set up a fully automated rotation cycle using widely‑available DevOps tools—Ansible for configuration, HashiCorp Vault for secrets management, and a lightweight scheduler (Cron or GitHub Actions) to trigger the process. By the end of the tutorial you will have a repeatable pipeline that rotates host keys on any target, updates known_hosts files for trusted clients, and records the change in an audit log.

What You’ll Learn

• Why regular ssh host key rotation matters for hybrid cloud security.
• How to generate, store, and distribute host keys safely.
• Step‑by‑step Ansible playbooks that work on on‑premise and cloud hosts.
• Integration with Vault for secret‑zero handling.
• How to schedule and verify rotations without service disruption.

Why rotate SSH host keys?

Host keys provide the cryptographic identity of a server. If an attacker gains read access to /etc/ssh/ssh_host_*_key, they can perform man‑in‑the‑middle attacks against any client that trusts the compromised key. Regular rotation reduces the window of exposure and satisfies compliance frameworks such as PCI‑DSS and NIST SP 800‑53, both of which list “key management” as a required control.

In a hybrid cloud, the risk is amplified because:

• On‑premise systems may run older OpenSSH versions that lack modern algorithms.
• Cloud‑provider metadata services sometimes expose host key fingerprints in console logs.
• Automation tools (e.g., Terraform, Cloud‑Init) may recreate instances, unintentionally re‑using old keys.

Prerequisites and assumptions

• SSH servers run OpenSSH ≥ 7.9 (the version shipped with Ubuntu 22.04, RHEL 9, etc.).
• Ansible ≥ 2.14 installed on a control node (official docs).
• HashiCorp Vault ≥ 1.13 accessible from the control node (official docs).
• Inventory that groups hosts by onprem and cloud variables.
• At least one user with sudo privileges on each target.

Architecture overview

The rotation pipeline consists of four logical components:

Setting up Vault for host key storage

Enable KV v2 secret engine

vault secrets enable -path=ssh-host-keys kv

Create a policy for rotation

path "ssh-host-keys/data/{{identity.entity.name}}/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

Attach the policy to the AppRole that Ansible will use (Vault AppRole docs).

Store an initial key (optional)

If you already have a key pair, import it so the first rotation replaces the existing value.

vault kv put ssh-host-keys/data/host01 \
  private_key="$(cat /etc/ssh/ssh_host_ed25519_key)" \
  public_key="$(cat /etc/ssh/ssh_host_ed25519_key.pub)"

Ansible playbook for rotation

Directory layout

.
├── inventory/
│   ├── hosts.yml
│   └── group_vars/
│       ├── onprem.yml
│       └── cloud.yml
├── playbooks/
│   └── rotate_ssh_keys.yml
└── roles/
    └── ssh_rotate/
        ├── tasks/
        │   └── main.yml
        └── templates/
            └── sshd_config.j2

inventory/hosts.yml (example)

all:
  children:
    onprem:
      hosts:
        srv01.example.com:
        srv02.example.com:
    cloud:
      hosts:
        i-0a1b2c3d4e5f6g7h8:
          ansible_host: 34.212.45.67

roles/ssh_rotate/tasks/main.yml

- name: Generate new ED25519 host key
  command: ssh-keygen -t ed25519 -f /tmp/ssh_host_ed25519_key -N ""
  args:
    creates: /tmp/ssh_host_ed25519_key
  delegate_to: localhost
  run_once: true

- name: Read generated keys
  slurp:
    src: "{{ item }}"
  register: key_contents
  loop:
    - /tmp/ssh_host_ed25519_key
    - /tmp/ssh_host_ed25519_key.pub
  delegate_to: localhost
  run_once: true

- name: Store private key in Vault
  community.hashicorp.vault_kv2_write:
    url: "{{ vault_addr }}"
    token: "{{ vault_token }}"
    mount_point: ssh-host-keys
    path: "{{ inventory_hostname }}"
    data:
      private_key: "{{ key_contents.results[0].content | b64decode }}"
      public_key: "{{ key_contents.results[1].content | b64decode }}"
  delegate_to: localhost
  run_once: true

- name: Deploy new private key to target
  copy:
    content: "{{ lookup('community.hashicorp.vault_kv2_read', 'ssh-host-keys/data/' + inventory_hostname).data.private_key }}"
    dest: /etc/ssh/ssh_host_ed25519_key
    owner: root
    group: root
    mode: '0600'
  become: true

- name: Deploy matching public key
  copy:
    content: "{{ lookup('community.hashicorp.vault_kv2_read', 'ssh-host-keys/data/' + inventory_hostname).data.public_key }}"
    dest: /etc/ssh/ssh_host_ed25519_key.pub
    owner: root
    group: root
    mode: '0644'
  become: true

- name: Ensure sshd uses the new key
  lineinfile:
    path: /etc/ssh/sshd_config
    regexp: '^HostKey /etc/ssh/ssh_host_ed25519_key'
    line: 'HostKey /etc/ssh/ssh_host_ed25519_key'
  become: true

- name: Restart sshd
  service:
    name: sshd
    state: restarted
  become: true

- name: Update known_hosts on bastion hosts
  delegate_to: "{{ item }}"
  known_hosts:
    name: "{{ inventory_hostname }}"
    key: "{{ lookup('community.hashicorp.vault_kv2_read', 'ssh-host-keys/data/' + inventory_hostname).data.public_key }}"
    path: /home/{{ bastion_user }}/.ssh/known_hosts
  loop: "{{ groups['bastion'] | default([]) }}"
  become: true

Running the playbook

ansible-playbook -i inventory/hosts.yml playbooks/rotate_ssh_keys.yml \
  --extra-vars "vault_addr=https://vault.example.com vault_token=$VAULT_TOKEN bastion_user=admin"

Pro Tip: Use Ansible Vault to encrypt the vault_token variable, then decrypt at runtime with ansible-playbook --ask-vault-pass.

Handling cloud‑specific nuances

Public‑cloud VMs often receive their SSH host keys from the image’s cloud‑init configuration. To prevent the image from overwriting the rotated key, add a cloud‑init snippet that disables automatic key generation.

AWS example (user‑data)

#cloud-config
ssh:
  emit_keys_to_console: false
  host_key:
    rsa: false
    ecdsa: false
    ed25519: true

For Azure, set disablePasswordAuthentication: true and ensure sshKeys: [] in the customData section. GCP uses the sshKeys metadata key; leave it empty to avoid injection.

Scheduling the rotation

Choose a scheduler that fits your operational model:

• Cron on the Ansible control node – simple, no extra cost.
• GitHub Actions or GitLab CI – good for teams already using version‑controlled pipelines.
• HashiCorp Nomad job with a periodic stanza – integrates with Vault token renewal.

Example cron entry (rotate every 90 days)

0 3 1 */3 * /usr/local/bin/ansible-playbook -i /opt/ssh-rotate/inventory/hosts.yml \
  /opt/ssh-rotate/playbooks/rotate_ssh_keys.yml \
  --extra-vars "@/opt/ssh-rotate/vars/vault.yml" >> /var/log/ssh-rotate.log 2>&1

Validation and rollback

1. Post‑rotation health check: run ssh -o StrictHostKeyChecking=no {{ inventory_hostname }} echo ok from a bastion host.
2. Audit log: Vault writes a versioned entry; keep the previous version for at least one rotation cycle.
3. Rollback procedure: retrieve the prior version with vault kv get -version=<N> ssh-host-keys/data/{{ inventory_hostname }} and re‑run the deployment tasks.

Tool comparison

Frequently Asked Questions

How often should SSH host keys be rotated?

There is no universal mandate, but many compliance frameworks recommend a 90‑day rotation cycle. Adjust the interval based on risk assessment and operational impact.

Will rotating the host key disconnect existing SSH sessions?

Active sessions remain open because the key is only checked during the initial handshake. New connections will use the updated key immediately after the sshd restart.

Can I rotate keys without restarting sshd?

OpenSSH supports reloading the configuration with systemctl reload sshd, but the new host key is only read on a full restart. Using reload will not replace the key in memory.

What algorithm should I use for new host keys?

ED25519 offers strong security with small key size and fast computation. RSA ≥ 4096 bits is acceptable but larger and slower. OpenSSH defaults to ED25519 as of version 7.6.

How do I ensure clients trust the new key?

Distribute the new public key to all trusted bastion or jump hosts and update their known_hosts files via the Ansible known_hosts module. For end‑users, consider a short‑lived warning banner until they accept the new fingerprint.

Is it safe to store private host keys in Vault?

Vault’s KV v2 engine encrypts data at rest and in transit. When combined with a tightly scoped AppRole and short‑lived tokens, it meets most security requirements for secret‑zero handling (Vault docs).

What if a rotation fails on a subset of hosts?

Design the pipeline to be idempotent: the playbook can be re‑run without side effects. Use Ansible’s --limit flag to target only the failed hosts, and invoke the rollback steps if necessary.

Next steps

1. Clone the repository structure provided in this guide to a version‑controlled location.
2. Configure Vault policies and generate an AppRole token for the Ansible control node.
3. Run the playbook manually on a test host, verify the new fingerprint, and commit the changes.
4. Integrate the cron or CI job, monitor the log output, and schedule the first production rotation.
5. Document the rollback procedure in your incident‑response runbook.

Official documentation

• Ansible Documentation
• HashiCorp Vault Docs
• ssh-keygen manual
• cloud‑init SSH module
• AWS Systems Manager Automation