Skip to main content

HashiCorp Vault Cluster Unseal

These steps are for a vault cluster that has been configured and started up, but is in an unsealed state.

Leader Initialization

The following will initialize a new vault cluster and return a set of unseal keys and a root token.

NOTE: The ca.crt file is privileged, You will need to run these commands as the root user.
Run the following to switch to the root user:

sudo -i

Go to the first node, and do these (as root):

# From an admin shell that can reach the VLAN:
export VAULT_ADDR="https://vault0204.ogsofttech.lan:8200"
export VAULT_CACERT="/opt/vault/tls/ca.crt"   # path on your admin box

# Initialize the cluster (choose your own shares/threshold)
vault operator init -key-shares=5 -key-threshold=3

NOTE: Use the fully-qualified hostname above, as it appears in the node's cert.

Once executed, the vault node will reply with 5 unseal keys and an initial root token.

Distribute each of these unseal keys to trusted admins, to store in offline password storage.

NOTE: Three (3) unseal keys are required to unseal the vault.

Use the initial root token to setup policies and auth.
Then, retire it.

Unseal the Leader

With the unseal keys from the initialized node (received above), we need to unseal its vault.

NOTE: We do this, while still as root, and on the same host that we got the keys from.

Now, unseal each node, by calling this command once each, for three of the five unseal keys:

NOTE: It will prompt you for the unseal key, each time you run it.

vault operator unseal

Initial Root Login

With the vault unsealed, we need to perform an initial login as root:

# Log in with the root token for initial setup tasks
vault login <root_token>

Once logged in, you can check the vault status with this:

vault status

If successful, you should see Initialized: true, Sealed: false, HA Enabled: true, and this node as leader.

The first node is online, and the cluster is up... sort of.

Each cluster member has auto-discovered a leader and established a RAFT quorum.

But, the other nodes are still not unsealed (since we did not configured auto-unseal).

Unseal Other Nodes

Similar to what you did, to unseal the first node, we will do the same to each member, below.

Switch to root on each node with:

sudo -i

Set exports for each node:

NOTE: Make sure that the vault_addr variable is pointing to the local node being unsealed, here.

export VAULT_ADDR="https://vault0205.ogsofttech.lan:8200";
export VAULT_CACERT="/opt/vault/tls/ca.crt"

NOTE: Use the fully-qualified hostname above, as it appears in the node's cert.

Now, unseal each node, by calling this command once each, for three of the five unseal keys:

NOTE: It will prompt you for the unseal key, each time you run it.

vault operator unseal

Check Status

Once you have initialized the leader node, and unsealed all nodes, we need to confirm that the cluster is good.

Run this:

NOTE: The vault_addr should be pointing to the leader, here.

# Set this if you are coming back to this page, and the environment value is not set...
export VAULT_ADDR="https://vault0204.ogsofttech.lan:8200";
# Run this to check status...
vault status

Confirm RAFT peers with this:

vault operator raft list-peers

NOTE: The above may only work on the current leader, because https.
We need to work through why this is, and solve it, so it can be run on any node.

When run, you will see something like this:

image.png

If healthy, you will see one node as leader, and the others as voting followers.

NOTE: Make sure each node you configured, is present.

There is a health check that can be performed, by calling this:

curl -s -H "X-Vault-Token: $VAULT_TOKEN" "$VAULT_ADDR/v1/sys/ha-status" | jq .

NOTE: Be sure that the VAULT_TOKEN and VAULT_ADDR environment variables are set.
Or, you can hardcode them with a minimal privilege user account.

When run, you will get a JSON list of nodes and their status.

Audit Logging

Create a folder for capturing audit logs on each node (leader and followers):

sudo mkdir -p /var/log/vault
sudo chown vault:vault /var/log/vault
sudo chmod 0750 /var/log/vault

Tell the leader node to store audit logs in the created folder:

vault audit enable file file_path=/var/log/vault/audit.log

Admin Policy

WARNING: The initial root token has broad privilege, and bypasses all policy checks, being meant for bootstrapping and emergencies.
You need to create an actual admin policy, so that proper security and auditing can occur. 

Here, we will setup an admin policy, so that we can stop using the initial root token.

Doing so, gives us many benefits:

  • Users created under the admin policy share the same privileges as the root user.
  • We can issue short-lived admin tokens.
  • We can audit all admin actions (audit logs will show policy=admin and the user, not just root).
  • We can later, reduced privileges, instead of everyone having anonymous root access.

NOTE: We only have to create the admin policy once.
We can do it on the current leader.

Create a file called: admin.hcl:

sudo nano /etc/vault.d/admin.hcl

NOTE: It doesn't matter where we generate the admin.hcl file, as the policy write command will pull it into the vault.
But for simplicity, we will create it on the leader node, so we have easy tracking of what setup steps have been done.

Give the admin policy file, this content:

# --- token management (mint/lookup/renew/revoke/roles) ---
path "auth/token/create"         { capabilities = ["update"] }
path "auth/token/create/*"       { capabilities = ["update"] }
path "auth/token/lookup"         { capabilities = ["read", "update"] }
path "auth/token/lookup-self"    { capabilities = ["read"] }
path "auth/token/renew"          { capabilities = ["update"] }
path "auth/token/renew-self"     { capabilities = ["update"] }
path "auth/token/revoke"         { capabilities = ["update"] }
path "auth/token/revoke-self"    { capabilities = ["update"] }
path "auth/token/roles"          { capabilities = ["list", "read"] }
path "auth/token/roles/*"        { capabilities = ["create","read","update","delete","list"] }

# --- policies (so admins can maintain policies without root) ---
path "sys/policies/acl"          { capabilities = ["list"] }
path "sys/policies/acl/*"        { capabilities = ["create","read","update","delete","list"] }

# --- core admin knobs (optional but typical) ---
path "sys/audit"                 { capabilities = ["read"] }
path "sys/audit/*"               { capabilities = ["create","read","update","delete","sudo"] }
path "sys/auth"                  { capabilities = ["read"] }
path "sys/auth/*"                { capabilities = ["create","read","update","delete","sudo"] }
path "sys/mounts"                { capabilities = ["read"] }
path "sys/mounts/*"              { capabilities = ["create","read","update","delete","sudo"] }

# --- give admin wide access to secrets during bootstrap (tighten later) ---
path "*"                         { capabilities = ["create","read","update","delete","list","sudo"] }

Once created, save and close.

Here are some notes about the above policy:

  • The top block allows an admin to perform all token management actions.
  • The middle block allows admins to manipulate policies.
  • The third block allows admins to perform other root actions.
  • The bottom block (path "*") allows access to all paths in the vault (like root).
    • capabilities = create/read/update/delete/list - These allow full browsing and editing.
    • capabilities = sudo - allows doing system-level actions like:
      • enabling audit devices,
      • enabling auth methods,
      • mounting new secrets engines,
      • etc.

So, the above admin policy is just like root, but allows us to have named users as admins.

Enable the admin policy with this:

vault policy write admin admin.hcl

Once written, the policy is stored in RAFT, and replicated to all nodes.

We can confirm the policy with this:

vault policy read admin

Now, any tokens with the admin policy, will function with the above administrative privileges.

Creating Admin Tokens

Once the admin policy exists, you can create administrative access tokens.

See this page for how to create admin and user tokens: Vault Token Administration

To protect tokens in transit, see this page for Response-Wrapped Tokens: Vault Wrapping Token

Back to top