Clustering HashiCorp Vault

Here are special instructions for setting up a vault cluster.

NOTE: See the regular setup page for other details: Hashicorp Vault Setup

DNS Resolution

Since the vault services will communicate with eachother over TLS, they will need certificates.
And as such, the certs will include hostnames.

So, open the /etc/hosts file of each vault host, and add entries, at the bottom of the file, for each instance and API host.

Here's an example list of entries for a cluster:

192.168.75.10 vault02api
192.168.75.21 vault0201
192.168.75.22 vault0202
192.168.75.23 vault0203
192.168.75.24 vault0204
192.168.75.25 vault0205
192.168.75.26 vault0206

Filesystem Changes

Raft Folder

The Vault service will be running Raft. So, it will need a folder for the Raft backend.

NOTE: This may mean that the folder /opt/vault/data is obsolete.
But, we will not worry about that, for now.
The 'data' folder was created by the installer as the FS location for a storage = 'file' backend.

Create the raft folder with these:

sudo mkdir /opt/vault/raft

sudo chmod 700 /opt/vault/raft

sudo chown vault:vault /opt/vault/raft

TLS Folder

The installer already created the TLS folder, to store certificates.
It is at: /opt/vault/tls

We will leave it as is.

Config Folder

The installer created a config folder at: /etc/vault.d

We need to bolster its permissions, as it may contain seal stanzas.

Update permissions of the config folder with these:

sudo chmod 0750 /etc/vault.d

sudo chown root:vault /etc/vault.d

Certificates

We will create certificates for each vault instance, and put them in the tls folder at: /opt/vault/tls

Follow instructions, here, to generate certificates for each host: Generate Certificates with Hashicorp Vault

NOTE: Be sure to do the following:
- Set the common name to the fully qualified name: ex: vault0204.ogsofttech.lan.
- Set the expiry to two years (17520 hours).
- Set set the SAN IP to the address of the host: 192.168.75.24

We will copy keys and certs with these:

# Drop your files in place
sudo cp vault.crt vault.key ca.crt /etc/vault.d/tls/

Once stored, we need to set permissions on the files, with these:

# Ownership: keep vault:vault
sudo chown vault:vault /etc/vault.d/tls/vault.crt /etc/vault.d/tls/vault.key /etc/vault.d/tls/ca.crt

# Permissions:
sudo chmod 0600 /etc/vault.d/tls/vault.key   # private key: only vault access
sudo chmod 0644 /etc/vault.d/tls/vault.crt   # server cert usually fine as world-readable
sudo chmod 0644 /etc/vault.d/tls/ca.crt      # CA cert usually fine as world-readable

Vault.HCL Changes

Open the config file at: /etc/vault.d/vault.hcl

We have created the following vault.hcl file for a cluster node that has an:

IP = 192.168.75.24
node_id = vault0204

# HTTPS listener
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/tls.crt"
  tls_key_file  = "/opt/vault/tls/tls.key"
}

# Vault Configuration for Clustering.
# Created on 20250901

ui            = true
cluster_name  = "prod-vault"
# recommended with Integrated Storage
disable_mlock = true

# Advertise addresses (MUST be correct & reachable)
# Define the external API address that clients will use to communicate with this Vault node.
# set per node
api_addr     = "https://vault-1.example.com:8200"
api_addr     = "https://192.168.75.24:8200"
# Define the internal address used by Vault nodes to communicate with this Vault node.
# Vault ignores the scheme of this URL, so it doesn't matter if http or https.
# set per node (node's intra-cluster IP:8201)
cluster_addr = "https://192.168.75.24:8201"

listener "tcp" {
  address                  = "0.0.0.0:8200"
  tls_disable              = 0
  tls_cert_file            = "/etc/vault.d/tls/vault.crt"
  tls_key_file             = "/etc/vault.d/tls/vault.key"
  tls_client_ca_file       = "/etc/vault.d/tls/ca.crt"
  # optional: require client certs from automation:
  # tls_require_and_verify_client_cert = "true"
}

# Updated to use raft backend (Integrated Storage).
storage "raft" {
  path    = "/opt/vault/raft"
# unique on each node  
  node_id = "vault0204"

  # Auto-join peers (preferred over manual joins)
  retry_join {
    leader_api_addr      = "https://vault-1.example.com:8200"
    leader_ca_cert_file  = "/etc/vault.d/tls/ca.crt"
  }
  retry_join {
    leader_api_addr      = "https://vault-2.example.com:8200"
    leader_ca_cert_file  = "/etc/vault.d/tls/ca.crt"
  }
  retry_join {
    leader_api_addr      = "https://vault-3.example.com:8200"
    leader_ca_cert_file  = "/etc/vault.d/tls/ca.crt"
  }
}

# Recommended health endpoint behavior for LBs
# (defaults are fine; LB should treat 200 as active, 429 as standby, 503 sealed)
# See /v1/sys/health docs for details.