# HashiCorp Vault



# Generate Certificates with Hashicorp Vault

Here are steps to generate SSL certificates using HashiCorp Vault as an Intermediate CA.

NOTE: Be sure that you've setup a vault instance as an Intermediate CA.  
See this page for how: [Vault as Intermediate CA](https://wiki.galaxydump.com/link/455)

Login to the web UI of your intermediate CA, such as: [https://vault02.ogsofttech.lan:8200/ui/](https://vault02.ogsofttech.lan:8200/ui/)  
If DNS is down, use this: [https://192.168.60.6:8200/ui/](https://192.168.60.6:8200/ui/)

For the latest intermediate CA url, see this page: [Vault Services](https://wiki.galaxydump.com/link/300)

Find the issuing role by navigating to Secrets/PKI/Roles.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/Ctq7LL1OQADcWAZf-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/Ctq7LL1OQADcWAZf-image.png)

Select the role, and click Generate Certificate:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/ZJgTJD3X6MrCO68T-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/ZJgTJD3X6MrCO68T-image.png)

Fill in the Common name as: router.ogsofttech.lan.

Set the TTL to 1 year (365 days).

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/w26rZjkslfBLo9UP-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/w26rZjkslfBLo9UP-image.png)

Click Generate, to create the key and certificate, and you’ll see this:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/0Lso9nwbaxRntk1h-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/0Lso9nwbaxRntk1h-image.png)

Download the private key as: router.ogsofttech.lan-key.pem

Download the certiticate as: router.ogsofttech.lan-cert.pem

Download the CA chain as: router.ogsofttech.lan-cabundle.pem

<p class="callout info">NOTE: We are calling the downloaded CA chain file a “ca bundle”.  
CA bundle is the standard naming convention for this file type.  
Specifically, a cert is often concatenated with the CA bundle that signed it, to create a chain certificate file.</p>

Now, you can copy the cert, ca bundle, and private key to the host, for usage.

If generating a pair for a linux host, you will need them as .crt and .key files.  
Follow this: [Converting PEM to crt and key](https://wiki.galaxydump.com/link/303 "https://wiki.galaxydump.com/link/303")

If generating a pair for an Nginx host, you will need

# Hashicorp Vault Setup

Here are steps for setting up a secrets store using Hashicorp Vault, on Ubuntu 24.

## References

Lots of steps were taken from here: [https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engine](https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engine)

## Server Setup

Before installing Vault, perform steps from this page, to setup the server: [Ubuntu Host Setup](https://wiki.galaxydump.com/link/245)

Since a typical vault cluster has limited internet visibility, it may be necessary to map in the local NTP server, to keep each node in sync.  
See this page for how to setup each host/VM to use the local private NTP server: [Ubuntu: Use Private NTP Server](https://wiki.galaxydump.com/link/457)

### Install Vault

Here are three lines to install vault.

First, we will setup the package source keys…

```
sudo wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
```

Now, add the package source entry…

```bash
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
```

Now, install vault…

```bash
sudo apt update && sudo apt install vault
```

### Vault User

The installer created a user called, vault, and the service runs under it.  
But, the default keys are restricted to root permissions.  
So, we need to fix access to certificates.

For this, we will create a group to make it easier to control and test access to certs and keys.

```bash
sudo groupadd pki
```

And, we will add the vault user to the group:

```bash
sudo usermod -a -G pki vault
```


#### Clustered Config

<p class="callout info">NOTE: If you are standing up a single node vault, skip this section.</p>

Work through the steps on this page, for clustering your vault servers: [Clustering HashiCorp Vault](https://wiki.galaxydump.com/link/448)

### Vault Configuration

<p class="callout info">NOTE: This is for a single vault node. If you are standing up a cluster, skip this section.</p>

Edit the config file at: /etc/vault.d/vault.hcl

Specifically, you want to set the tls cert/key paths to your own domain certs.

As well, you may want to limit the listener to just a single address, since the default is for all adapters on the host.

Once done, save the config update.

#### Continue Setup

Once installed and configured, we need to enable and start the vault services, with these lines:

```bash
sudo systemctl daemon-reload
sudo systemctl enable vault
sudo systemctl start vault
sudo systemctl status vault
```

You should be able to verify that the vault service is active by call this:

[https://&lt;host-ip&gt;:8200/v1/sys/seal-status](https://192.168.60.101:8200/v1/sys/seal-status "https://192.168.60.101:8200/v1/sys/seal-status")

<p class="callout info">NOTE: Use your VM’s fully-qualified name (or IP address), in the above.</p>

If successful, the above call will return something like this:

```json
{
  "type":"shamir",
  "initialized":false,
  "sealed":true,
  "t":0,
  "n":0,
  "progress":0,
  "nonce":"",
  "version":"1.17.6",
  "build_date":"2024-09-24T19:48:40Z",
  "migration":false,
  "recovery_seal":false,
  "storage_type":"file"
}
```

### Cluster Unseal

If you're standing up a cluster, see this page: [HashiCorp Vault Cluster Unseal](https://wiki.galaxydump.com/link/449)

If you're standing up a single-node vault instance, see this page: [Vault Single-Node Unseal](https://wiki.galaxydump.com/link/452)

### Administrative Setup

Once the vault is unsealed, you need to setup auditing and administrative policies.

See this page for how: [Vault Administrative Setup](https://wiki.galaxydump.com/link/453#bkmrk-page-title)

### Creating Users (Access Tokens)

Once the vault cluster is setup, you need to establish some administrative users.

See this page for how to create admin and user tokens: [Vault Token Administration](https://wiki.galaxydump.com/link/451)

To protect tokens in transit, see this page for Response-Wrapped Tokens: [Vault Wrapping Token](https://wiki.galaxydump.com/link/450)

### Login

Once your vault is unsealed, you can log into its UI.

Log into its UI, here: [http://&lt;host-ip&gt;:8200/](http://192.168.60.5:8200/ "http://192.168.60.5:8200/")

You will be prompted to enter your root token, here:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/fCJSlTXqhKlvOLbU-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/fCJSlTXqhKlvOLbU-image.png)

If successful, you will see the root user’s dashboard:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/8RcEWTlkuNOheZ78-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/8RcEWTlkuNOheZ78-image.png)

### Creating the First Secret Store

Now, you should create a secrets engine, to store things.  
The one that makes the most sense, first, is to make a KV store.  
You should be able to make a kv secrets store (actually, a kv-v2 store) from the dashboard:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/IJ5OkCQNHCb7zzMP-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/IJ5OkCQNHCb7zzMP-image.png)

The above will create the secrets store at the path: /kv  
You can change this if needed.

Once created, you will see the new empty store:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/Hl1aX8MoCi5VMGQk-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/Hl1aX8MoCi5VMGQk-image.png)

### Using Vault as a CA

Follow steps on this page, to setup a vault instance as a Root CA: [Vault as Root CA](https://wiki.galaxydump.com/link/454)

<p class="callout info">NOTE: You never want to directly sign certificates with a Root CA.  
If you do, there is no way to revoke the root CA, without collapsing all PKI chains (by revoking the root CA certificate).  
However. You can create an Intermediate CA that does signings.  
And, that Intermediate CA can be revoked, without losing the root CA certificate.</p>

<p class="callout info">So, we will create an Intermediate CA, to do actual signings.  
And, the Intermediate CA's certificate will be signed by the Root CA, just before offlining the Root CA.</p>

Once a root CA is setup, you can follow this page, to setup an Intermediate CA,  
that will do actual signings: [Vault as Intermediate CA](https://wiki.galaxydump.com/link/455)

See this page for how to generate certificates: [Generate Certificates with Hashicorp Vault](https://wiki.galaxydump.com/link/433)

# Clustering HashiCorp Vault

Here are special instructions for setting up a vault cluster.

<p class="callout info">NOTE: See the regular setup page for other details: [Hashicorp Vault Setup](https://wiki.galaxydump.com/link/434#bkmrk-page-title)</p>

#### DNS Resolution

Since the vault services will communicate with eachother over TLS, they will need certificates.  
And as such, the certs will include hostnames.

So, open the /etc/hosts file of each vault host, and add entries, at the bottom of the file, for each instance and API host.

Here's an example list of entries for a cluster:

```
192.168.75.10 vault02api
192.168.75.21 vault0201
192.168.75.22 vault0202
192.168.75.23 vault0203
192.168.75.24 vault0204
192.168.75.25 vault0205
192.168.75.26 vault0206
```

#### Filesystem Changes

##### Raft Folder

The Vault service will be running Raft. So, it will need a folder for the Raft backend.

<p class="callout info">NOTE: This may mean that the folder /opt/vault/data is obsolete.  
But, we will not worry about that, for now.  
The 'data' folder was created by the installer as the FS location for a storage = 'file' backend.</p>

Create the raft folder with these:

```
sudo mkdir /opt/vault/raft
```

```
sudo chmod 700 /opt/vault/raft
```

```
sudo chown vault:vault /opt/vault/raft
```

TLS Folder

The installer already created the TLS folder, to store certificates.  
It is at: /opt/vault/tls

We will leave it as is.

Config Folder

The installer created a config folder at: /etc/vault.d

We need to bolster its permissions, as it may contain seal stanzas.

Update permissions of the config folder with these:

```
sudo chmod 0750 /etc/vault.d
```

```
sudo chown root:vault /etc/vault.d
```

#### Firewall Rules

Update the local firewall rules for each vault host, to allow 8200 and 8201 access.

```bash
sudo ufw allow 8200
```

```bash
sudo ufw allow 8201
```

#### Certificates

We will create certificates for each vault instance, and put them in the tls folder at: /opt/vault/tls

Follow instructions, here, to generate certificates for each host: [Generate Certificates with Hashicorp Vault](https://wiki.galaxydump.com/link/433)

<p class="callout info">NOTE: Be sure to do the following:  
- Set the common name to the fully qualified name: ex: vault0204.ogsofttech.lan.  
- Set the expiry to two years (17520 hours).  
- Set set the SAN IP to the address of the host: 192.168.75.24</p>

Copy the CA certificate bundle (CA chain of issuer and root CA) into: /opt/vault/tls/ca.crt

<p class="callout info">NOTE: The ca.crt file should be the certificate bundle of issuer CA cert and root CA certificate.  
These should be bundled (root + intermediate) in the ca.crt file, as a concatenated PEM.</p>

Concatenate the vault service certificate with the issuer CA cert as a concatenated PEM.

Paste the certificate file (vault service + issuer CA cert) into: /opt/vault/tls/vault.crt

<p class="callout info">NOTE: The vault.crt file should include the leaf certificate (of the node) plus the signing intermediate.</p>

Copy the vault service private key into: /opt/vault/tls/vault.key

Once key and certs are stored, we need to set permissions on the files, with these:

```bash
# Ownership: keep vault:vault
sudo chown vault:vault /opt/vault/tls/vault.crt /opt/vault/tls/vault.key /opt/vault/tls/ca.crt

# Permissions:
# private key: only vault access
sudo chmod 0600 /opt/vault/tls/vault.key
# server cert usually fine as world-readable
sudo chmod 0644 /opt/vault/tls/vault.crt
# CA cert usually fine as world-readable
sudo chmod 0644 /opt/vault/tls/ca.crt
```

Here's a quick sanity check, to verify the certificates on a host:

```bash
# Verify the server chain file against your trust bundle
openssl verify -CAfile /opt/vault/tls/ca.crt /opt/vault/tls/vault.crt

# See what the gateway/clients will see
openssl s_client -connect vault0204:8200 -showcerts -verify_return_error \
  -CAfile /opt/vault/tls/ca.crt </dev/null
```

#### Vault.HCL Changes

Now, we need to create a config file for each vault service (vault.hcl).

There will be some tailoring required for each host.  
So, pay attention to the notes.

Open the vault config file at: /etc/vault.d/vault.hcl

```bash
sudo nano /etc/vault.d/vault.hcl
```

Below is a config file for a single node in a vault cluster.

<p class="callout info">NOTE: You will need to change the following lines for each host:  
 api\_addr  
 cluster\_addr  
 node\_id</p>

```json
# Vault Configuration for Clustering.
# Created on 20250901

ui            = true
cluster_name  = "vaultcluster2"
# recommended with Integrated Storage
disable_mlock = true

# Advertise addresses (MUST be correct & reachable)
# Define the external API address that clients will use to communicate with this Vault node.
# set per node
api_addr     = "https://vault0204.ogsofttech.lan:8200"
# Define the internal address used by Vault nodes to communicate with this Vault node.
# Vault ignores the scheme of this URL, so it doesn't matter if http or https.
# set per node (node's intra-cluster IP:8201)
cluster_addr = "https://192.168.75.24:8201"

listener "tcp" {
  address                  = "0.0.0.0:8200"
  tls_disable              = 0
  tls_cert_file            = "/opt/vault/tls/vault.crt"
  tls_key_file             = "/opt/vault/tls/vault.key"
  tls_client_ca_file       = "/opt/vault/tls/ca.crt"
  # optional: require client certs from automation:
  # tls_require_and_verify_client_cert = "true"
}

# Updated to use raft backend (Integrated Storage).
storage "raft" {
  path    = "/opt/vault/raft"
# unique on each node  
  node_id = "vault0204"

  # Auto-join peers (preferred over manual joins)
  retry_join {
    leader_api_addr      = "https://vault0204.ogsofttech.lan:8200"
    leader_ca_cert_file  = "/opt/vault/tls/ca.crt"
  }
  retry_join {
    leader_api_addr      = "https://vault0205.ogsofttech.lan:8200"
    leader_ca_cert_file  = "/opt/vault/tls/ca.crt"
  }
  retry_join {
    leader_api_addr      = "https://vault0206.ogsofttech.lan:8200"
    leader_ca_cert_file  = "/opt/vault/tls/ca.crt"
  }
}

# Recommended health endpoint behavior for LBs
# (defaults are fine; LB should treat 200 as active, 429 as standby, 503 sealed)
# See /v1/sys/health docs for details.

```

### Service File (Systemd Unit)

With folders fixed, certs and config defined, we need to configure the vault service for operation.

To do so, we need to identify the service file that the installer created, with this:

```bash
sudo systemctl cat vault.service
```

As of 20250803, the installer puts the systemd unit file, here:

```
/usr/lib/systemd/system/vault.service
```

We need to open it and update it:

```
sudo nano /usr/lib/systemd/system/vault.service
```

Once updated, save and close the systemd file.

Return to the generic vault setup page: [https://wiki.galaxydump.com/link/434#bkmrk-continue-setup](https://wiki.galaxydump.com/link/434#bkmrk-continue-setup)

# HashiCorp Vault Cluster Unseal

These steps are for a new vault cluster that has been configured and started up, but is in an unsealed state.

For an existing cluster with unsealed nodes, see this page: [Handling Vault Node Restart](https://wiki.galaxydump.com/link/458)

### Leader Initialization

The following will initialize a new vault cluster and return a set of unseal keys and a root token.

<p class="callout info">NOTE: The ca.crt file is privileged, You will need to run these commands as the root user.  
Run the following to switch to the root user:</p>

```bash
sudo -i
```

Go to the first node, and do these (as root):

```bash
# From an admin shell that can reach the VLAN:
export VAULT_ADDR="https://vault0204.ogsofttech.lan:8200"
export VAULT_CACERT="/opt/vault/tls/ca.crt"   # path on your admin box

# Initialize the cluster (choose your own shares/threshold)
vault operator init -key-shares=5 -key-threshold=3
```

<p class="callout info">NOTE: Use the fully-qualified hostname above, as it appears in the node's cert.</p>

Once executed, the vault node will reply with 5 unseal keys and an initial root token.

<p class="callout warning">Distribute each of these unseal keys to trusted admins, to store in offline password storage.</p>

<p class="callout info">NOTE: Three (3) unseal keys are required to unseal the vault.</p>

<p class="callout info">Use the initial root token to setup policies and auth.  
Then, retire it.</p>

### Unseal the Leader

With the unseal keys from the initialized node (received above), we need to unseal its vault.

<p class="callout info">NOTE: We do this, while still as root, and on the same host that we got the keys from.</p>

Now, unseal each node, by calling this command once each, for three of the five unseal keys:

<p class="callout info">NOTE: It will prompt you for the unseal key, each time you run it.</p>

```bash
vault operator unseal
```

### Initial Root Login

With the vault unsealed, we need to perform an initial login as root:

```bash
# Log in with the root token for initial setup tasks
vault login <root_token>
```

Once logged in, you can check the vault status with this:

```bash
vault status
```

If successful, you should see `Initialized: true`, `Sealed: false`, `HA Enabled: true`, and this node as leader.

The first node is online, and the cluster is up... sort of.

Each cluster member has auto-discovered a leader and established a RAFT quorum.

But, the other nodes are still not unsealed (since we did not configured auto-unseal).

### Unseal Other Nodes

Similar to what you did, to unseal the first node, we will do the same to each member, below.

Switch to root on each node with:

```bash
sudo -i
```

Set exports for each node:

<p class="callout info">NOTE: Make sure that the vault\_addr variable is pointing to the local node being unsealed, here.</p>

```bash
export VAULT_ADDR="https://vault0205.ogsofttech.lan:8200";
export VAULT_CACERT="/opt/vault/tls/ca.crt"
```

<p class="callout info">NOTE: Use the fully-qualified hostname above, as it appears in the node's cert.</p>

Now, unseal each node, by calling this command once each, for three of the five unseal keys:

<p class="callout info">NOTE: It will prompt you for the unseal key, each time you run it.</p>

```bash
vault operator unseal
```

### Check Status

Once you have initialized the leader node, and unsealed all nodes, we need to confirm that the cluster is good.

Run this:

<p class="callout info">NOTE: The vault\_addr should be pointing to the leader, here.</p>

```bash
# Set this if you are coming back to this page, and the environment value is not set...
export VAULT_ADDR="https://vault0204.ogsofttech.lan:8200";
# Run this to check status...
vault status
```

Confirm RAFT peers with this:

```bash
vault operator raft list-peers
```

<p class="callout warning">NOTE: The above may only work on the current leader, because https.  
We need to work through why this is, and solve it, so it can be run on any node.</p>

When run, you will see something like this:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/HCX5VSa7OOY5gn5s-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/HCX5VSa7OOY5gn5s-image.png)

If healthy, you will see one node as leader, and the others as voting followers.

<p class="callout info">NOTE: Make sure each node you configured, is present.</p>

There is a health check that can be performed, by calling this:

```bash
curl -s -H "X-Vault-Token: $VAULT_TOKEN" "$VAULT_ADDR/v1/sys/ha-status" | jq .
```

<p class="callout info">NOTE: Be sure that the VAULT\_TOKEN and VAULT\_ADDR environment variables are set.  
Or, you can hardcode them with a minimal privilege user account.</p>

When run, you will get a JSON list of nodes and their status.

# Vault Wrapping Tokens

When creating access tokens for HashiCorp Vault, you always want to prevent them from falling into the wrong hands, or showing up as clear-text in command line history, logs and audit trains.

To reduce the chance of tokens being passed in the clear, you can create a new user token in a response-wrapped state.  
This allows the token to be given to a user, over chat, or text, without much concern.

The response-wrapped token has a very short lifetime, and can only be redeemed once.

Here's how to create a response-wrapped token:

```bash
vault token create -role=admin -orphan -wrap-ttl=5m -format=json
```

The command response will include the wrapping token, like this:

```json
{
  "request_id": "",
  "lease_id": "",
  "lease_duration": 0,
  "renewable": false,
  "data": null,
  "warnings": null,
  "wrap_info": {
    "token": "hvs.tH5Wn8bD3eJxvMq1iP7F",
    "accessor": "PNowpcQP0jJ9Jpz06o2oueYW",
    "ttl": 300,
    "creation_time": "2025-09-04T02:26:49.859843035Z",
    "creation_path": "auth/token/create/admin",
    "wrapped_accessor": "5KlfcQwbo05Ej37YOqBJnfHM"
  }
}
```

The wrap\_info/token property is what you give to the user.

The user can then, redeem their access token by submitting the wrapping token, like this:

```
vault unwrap hvs.tH5Wn8bD3eJxvMq1iP7F
```

Vault will respond with the real token, and revoke further usage of the wrapping token.

<p class="callout info">NOTE: The wrapping token has an expiry. If exceeded, the user will need to request another.</p>

### Wrapped Token Benefits

Using wrapped tokens, prevents exposure in shell history or logs.

Delivery is safer, as you can drop the short-lived wrapper into a config management pipeline or paste in a chat.

The Vault audit log records that the wrapping token was created and who unwrapped it.

# Vault Token Administration

Here are notes on access token administration.

Be sure that you've already setup an administrative policy in your vault cluster.  
See this page for how: [Vault Administrative Setup](https://wiki.galaxydump.com/link/453#bkmrk-page-title)

### Creating Admin Tokens

Once the admin policy exists, you can create administrative access tokens.

To do so, log into the leader note, with this:

```bash
# Use the root token...
vault login <root_token>
```

You can create a simple admin token that lasts 24 hours:

```bash
# Create an admin token that lasts 24 hours...
vault token create -policy=admin -ttl=24h
```

The above will issue short-lived admin tokens, so that the vault is more protected.  
These can be minted by an automated script, that issues token as needed.

### Traditional Admin

A more traditional administrative token needs to be longer-lived than a 24 hour token.

So, we will create a token role for admins.

This will create a periodic, orphan role so that admin tokens can be renewed forever (until revoked), and aren't tied to a parent token:

<p class="callout info">NOTE: You will need to be logged into the vault leader (vault login &lt;token&gt;).</p>

```bash
vault write auth/token/roles/admin \
  allowed_policies="admin,default" \
  orphan=true \
  renewable=true \
  period=720h
```

The above token role, can create tokens that last 30 days, and can be renewed, without changing the token.

With the above admin role, you can create admin tokens of a more traditional lifetime, with this:

```bash
vault token create -role=admin -orphan -format=json
```

The above command will create admin tokens that can be given to your ops group, for longer-lived administrative access.

<p class="callout info">NOTE: Be sure to save the client\_token field in a secure place, such as a private password manager.</p>

### Token Renewal

Before a token expires, you can renew it with this command:

```bash
VAULT_TOKEN=<admin_token> vault token renew
```

<p class="callout info">NOTE: The above can be automated, if the admin token is used by scripts or services.  
(Automate with a small systemd timer/cron on a secure host or in your gateway.)  
</p>

### Creating Other Admins

You can use an admin token to create other admins, like this:

```bash
# Another admin token (same role)
vault login <admin_token>
vault token create -role=admin -orphan -format=json
```

### Creating Tokens for other Groups

Once you have other roles established as policies, you can create tokens for users, in those roles/policies, like this:

```bash
vault token create -policy=kv-readers -ttl=8h
```

### Response-Wrapped Tokens

See this page for Response Wrapped Tokens: [Vault Wrapping Token](https://wiki.galaxydump.com/link/450)

# Vault Single-Node Unseal

Here are instructions on how to unseal a single-node vault cluster.

### Vault CLI Comms

For the vault command to communicate with the running service, we must set an env variable for it:

```bash
export VAULT_ADDR=https://your_domain:8200
```

<p class="callout info">NOTE: You may have to set the above to http, if you haven’t given vault a valid ssl cert, yet.</p>

### Initialize the Vault

In order for the vault service to manage secrets, you must initialize its store.  
To do this, you need to run the following:

```bash
vault operator init -key-shares=3 -key-threshold=2
```

You can change the key shares and threshold based on your risk profile.

The vault CLI will respond with something like this:

```bash
Unseal Key 1: eZcJeydRrqeSMZ1zTN+VVll9TFT2KvJy7VlnxUgtvuz5
Unseal Key 2: ntmqCKq8rgNnKT1YSLCjVmCCZBAA3NwUeqxIyRpYD4Wm
Unseal Key 3: 3FK1+Hsorh4p8/L9mki3VskaEU2eQhLqGOI/pJkTHMbx

Initial Root Token: s.hY0ieybfDqCadz7JpL88uO3x
```

Save these values in a secure location, as this is the only time you will receive them.

Now, you can run the vault status command, and see it has changed to Initialized=true.

Run this to get vault status:

```bash
vault status
```

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/4XqZ3L6xD4ZuobAw-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/4XqZ3L6xD4ZuobAw-image.png)

Now, you need to unseal the vault, so it can be used.

Notice the vault status showed an unseal progress of 0/2.  
This means, that at least two more unseal tokens must be submitted for it to be available for access.

Enter enough tokens to unlock your vault with this command, and paste in one of the tokens when prompted:

```bash
vault operator unseal
```

Once the unseal threshold has been met, the Sealed state will become ‘false’.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/ckVlpmyoclmmlBcy-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/ckVlpmyoclmmlBcy-image.png)

Your vault is available for access.

# Vault Administrative Setup

Once you have a vault instance or cluster unsealed, you can setup auditing, and administrative policies, with these instructions.

See this page for how to setup a vault instance or cluster: [Hashicorp Vault Setup](https://wiki.galaxydump.com/link/434)

### Audit Logging

Create a folder for capturing audit logs on each node (leader and followers):

```bash
sudo mkdir -p /var/log/vault
sudo chown vault:vault /var/log/vault
sudo chmod 0750 /var/log/vault
```

Tell the leader node to store audit logs in the created folder:

```bash
vault audit enable file file_path=/var/log/vault/audit.log
```

### Admin Policy

<p class="callout warning">WARNING: The initial root token has broad privilege, and bypasses all policy checks, being meant for bootstrapping and emergencies.  
You need to create an actual admin policy, so that proper security and auditing can occur. </p>

Here, we will setup an admin policy, so that we can stop using the initial root token.

Doing so, gives us many benefits:

- Users created under the admin policy share the same privileges as the root user.
- We can issue short-lived admin tokens.
- We can audit all admin actions (audit logs will show policy=admin and the user, not just root).
- We can later, reduced privileges, instead of everyone having anonymous root access.

<p class="callout info">NOTE: We only have to create the admin policy once.  
We can do it on the current leader.</p>

Create a file called: admin.hcl:

```bash
sudo nano /etc/vault.d/admin.hcl
```

<p class="callout info">NOTE: It doesn't matter where we generate the admin.hcl file, as the policy write command will pull it into the vault.  
But for simplicity, we will create it on the leader node, so we have easy tracking of what setup steps have been done.</p>

Give the admin policy file, this content:

```json
# --- token management (mint/lookup/renew/revoke/roles) ---
path "auth/token/create"         { capabilities = ["update"] }
path "auth/token/create/*"       { capabilities = ["update"] }
path "auth/token/lookup"         { capabilities = ["read", "update"] }
path "auth/token/lookup-self"    { capabilities = ["read"] }
path "auth/token/renew"          { capabilities = ["update"] }
path "auth/token/renew-self"     { capabilities = ["update"] }
path "auth/token/revoke"         { capabilities = ["update"] }
path "auth/token/revoke-self"    { capabilities = ["update"] }
path "auth/token/roles"          { capabilities = ["list", "read"] }
path "auth/token/roles/*"        { capabilities = ["create","read","update","delete","list"] }

# --- policies (so admins can maintain policies without root) ---
path "sys/policies/acl"          { capabilities = ["list"] }
path "sys/policies/acl/*"        { capabilities = ["create","read","update","delete","list"] }

# --- core admin knobs (optional but typical) ---
path "sys/audit"                 { capabilities = ["read"] }
path "sys/audit/*"               { capabilities = ["create","read","update","delete","sudo"] }
path "sys/auth"                  { capabilities = ["read"] }
path "sys/auth/*"                { capabilities = ["create","read","update","delete","sudo"] }
path "sys/mounts"                { capabilities = ["read"] }
path "sys/mounts/*"              { capabilities = ["create","read","update","delete","sudo"] }

# --- give admin wide access to secrets during bootstrap (tighten later) ---
path "*"                         { capabilities = ["create","read","update","delete","list","sudo"] }

```

Once created, save and close.

Here are some notes about the above policy:

- The top block allows an admin to perform all token management actions.
- The middle block allows admins to manipulate policies.
- The third block allows admins to perform other root actions.
- The bottom block (path "\*") allows access to all paths in the vault (like root). 
    - capabilities = create/read/update/delete/list - These allow full browsing and editing.
    - capabilities = sudo - allows doing system-level actions like: 
        - enabling audit devices,
        - enabling auth methods,
        - mounting new secrets engines,
        - etc.

So, the above admin policy is just like root, but allows us to have named users as admins.

Enable the admin policy with this:

```bash
vault policy write admin admin.hcl
```

Once written, the policy is stored in RAFT, and replicated to all nodes.

We can confirm the policy with this:

```
vault policy read admin
```

Now, any tokens with the admin policy, will function with the above administrative privileges.

### Creating Admin Tokens

Once the admin policy exists, you can create administrative access tokens.

See this page for how to create admin and user tokens: [Vault Token Administration](https://wiki.galaxydump.com/link/451)

To protect tokens in transit, see this page for Response-Wrapped Tokens: [Vault Wrapping Token](https://wiki.galaxydump.com/link/450)

# Vault as Root CA

Here are steps you can follow to setup a vault instance as a Root CA.

<p class="callout info">NOTE: This page assumes that you have created a single-node vault instance to serve as your Root CA.  
See this page for how to do that: [Hashicorp Vault Setup](https://wiki.galaxydump.com/link/434)</p>

<p class="callout info">NOTE: These steps will create a root CA with one signing key.  
You should create an intermediate CA, as well, that will perform the actual signing of certs.  
This will allow you to offline this root CA, once the intermediate CA is up.  
</p>

### Starting the PKI Engine

<p class="callout info">NOTE: From here down, is steps for creating a Root CA.  
If you are setting up an Intermediate CA, skip to ‘Configure Intermediate CA’.</p>

For Vault to serve as a root CA, you have to add the PKI secrets engine.

To do this, enable the pki secrets engine:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/BJEMdmaa924Dxt4l-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/BJEMdmaa924Dxt4l-image.png)

For a Root CA, set the Max lease time to as long as possible, as the root CA will be offlined after generation.

Once initialized, it will look like this:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/7UXBd9LqRWxNChmF-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/7UXBd9LqRWxNChmF-image.png)

For a root CA, click Configure PKI to begin setup.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/5QsHHgxFSMzVfK24-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/5QsHHgxFSMzVfK24-image.png)

Choose the Generate Root option.

Set Type to Internal.

Set your Common name. Usually, this is a domain with a private TLD suffix.

Set the issuer name.

Set a TTL that is maxed out: 87600 hours.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/35DUBNK47kciPGoZ-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/35DUBNK47kciPGoZ-image.png)

Scroll down and fill in the issuer URLs, matching the origin for your root CA host:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/JbkGlKLJpFTuB756-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/JbkGlKLJpFTuB756-image.png)

When finished, click Done, to generate the root CA key and certificate.

You will see a page like this:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/3WbleXpXtzVMm7aI-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/3WbleXpXtzVMm7aI-image.png)

Copy out the root CA certificate, and save it to a file, named: `ogsofttech.lan_ca.crt`

Install it on all machines of the local network, so they will recognized the signed SSL certs of hosts.

### Current Intranet Root CA Certificate

The current root CA for the local intranet can be found in the secure share at this path:

```
\SecureShare git\oga\ogsofttech.lan\rootCA
```

<p class="callout info">NOTE: This certificate should be installed on any host that will consume services signed by the root ca.</p>

See this page for how to install it on an Ubuntu host: [How to Add Root CAs to Ubuntu](https://wiki.galaxydump.com/link/432)

### Root CA Rotation

To make things easier, when it comes time to rotate your root CA key, add a role, now.

Click PKI.

Click Roles.

Click Create Role.

Give the new role a name that you will recognize as the root CA rotation role: CA\_rotation\_role

Leave the rest of the fields empty, and click Create.

Now, you have a working root CA key/cert for your network.

We will use it to sign the certificate of an Intermediate CA, that will do all the actual work.

And, we will offline the root CA.

# Vault as Intermediate CA

Here are instructions for setting up a vault instance as an Intermediate CA.

<p class="callout info">NOTE: This page assumes that you have created a single-node vault instance to serve as your Root CA.  
See this page for how to do that: [Hashicorp Vault Setup](https://wiki.galaxydump.com/link/434)</p>

<p class="callout info">NOTE: Always create the Intermediate CA as a separate instance from your Root CA.  
This allows you to offline the Root CA, to uphold the integrity of your root CA certificate.</p>

To properly sign certificates, you will want to generate an Intermediate CA that will do all the actual issuing.  
And, you will offline the root CA, after the Intermediate CA is usable.

See this page if you need to setup a vault instance as a Root CA: [Vault as Root CA](https://wiki.galaxydump.com/link/454)

### Configure Intermediate CA

For Vault to serve as an Intermediate CA, you have to add the PKI secrets engine.

To do this, enable the pki secrets engine:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/mXT8nc47Ja3TXmiF-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/mXT8nc47Ja3TXmiF-image.png)

For an Intermediate CA, set the Max lease time to 43800 hours, and enable the engine.

Once initialized, it will look like this:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/ZutfwCBRaLggp7yy-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/ZutfwCBRaLggp7yy-image.png)

For an Intermediate CA, click Configure PKI to begin setup.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/LB4ciTg9hATmaz9k-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/LB4ciTg9hATmaz9k-image.png)

Choose the Generate intermediate CSR option, and fill in the blanks.

Set Type to Internal.

Set your Common name to your domain with a suffix that it’s an Intermediate CA, like this:

```
ogsofttech.lan Int CA 01
```

Click Generate, to make the CSR.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/8u6Ai3jPsxRNzqew-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/8u6Ai3jPsxRNzqew-image.png)

Paste the CSR onto the clipboard, and save it as a file: `pki_intermediate.csr`.

Open the web UI for your root CA, and navigate to Secrets/PKI/Issuers, and click Sign Intermediate.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/T8DwcOLp0OuEd908-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/T8DwcOLp0OuEd908-image.png)

Paste the CSR content into the CSR field.

Set the domain name as the common name: ogsofttech.lan

Select pem\_bundle from the Format dropdown:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/bsh8zbtSHeyd0f7D-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/bsh8zbtSHeyd0f7D-image.png)

Click Sign, to sign the CSR from the Intermediate CA.

Once signed, you’ll see this:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/A0CNNYtP5Y9IuvEO-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/A0CNNYtP5Y9IuvEO-image.png)

It contains the signed certificate for the Intermediate CA, the cert of the issuing CA, and the CA chain cert.

<p class="callout info">NOTE: Save all three in a known place, as this is the only time these will be available.</p>

Download the certificate as: `ogsofttech.lan-intca01.cert.pem`

Open the UI for the Intermediate CA, and navigate to the Import a CA:

<p class="callout info">NOTE: You get to this, by clicking on the Issuers tab, and clicking Import.</p>

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/nQN8tPwbVtk9I5Aa-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/nQN8tPwbVtk9I5Aa-image.png)

Upload the certificate that you saved to: `ogsofttech.lan-intca01.cert.pem`

Click Import Issuer.

The Issuers tab will show two entries, one for the root, and one for the Int CA:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/Xs71GkhQcFN32fe0-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/Xs71GkhQcFN32fe0-image.png)

## Issuing Role

Before you can issue certificates to hosts and such, you need to create an issuer role.

To do this, we need to identify the issuer’s Guid.  
We can get this by clicking the terminal button, and running this:

```bash
read -field=default pki/config/issuers
```

Note the Guid that was returned.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/RH1lhiMNltBT4skL-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/RH1lhiMNltBT4skL-image.png)

Now, go to Secrets/PKI/Roles, and create a new role.

Give the role a name in this format: ogsofttech-dot-lan

Unclick the Use Default Issuer, and select the Guid that matches what we found above.

Set the TTL to 43800 hours.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/lcE0BNnU8PL1PnT6-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/lcE0BNnU8PL1PnT6-image.png)

At the bottom of the form…

In the Allowed Domains field, add our domain: ogsofttech.lan.

Check the “Allow Subdomains” checkbox.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/DV4mRbtOBmSWzvHl-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/DV4mRbtOBmSWzvHl-image.png)

And, click Create, to make our issuing role.

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/UiQVCPuZvmYtRzHP-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/UiQVCPuZvmYtRzHP-image.png)

Now, you have a role that can create and sign certificates.

If not already done, you can offline your root CA, as it needs to be kept safe, and only needed to create a new Intermediate CA.

See this page for how to generate certificates: [Generate Certificates with Hashicorp Vault](https://wiki.galaxydump.com/link/433)

# Handling Vault Node Restart

Each time a Vault node restarts, you will have to unseal it, so that it can participate in the cluster.  
Here are steps to do that.

Switch to root on the node with:

```bash
sudo -i
```

Set exports for the node:

<p class="callout info">NOTE: Make sure that the vault\_addr variable is pointing to the local node being unsealed, here.</p>

```bash
export VAULT_ADDR="https://vault0205.ogsofttech.lan:8200";
export VAULT_CACERT="/opt/vault/tls/ca.crt"
```

<p class="callout info">NOTE: Use the fully-qualified hostname above, as it appears in the node's cert.</p>

Now, unseal each node, by calling this command once each, for three of the five unseal keys:

<p class="callout info">NOTE: It will prompt you for the unseal key, each time you run it.</p>

```bash
vault operator unseal
```

Once unsealed, you can verify cluster membership with this:

```bash
vault operator raft list-peers
```

When run, you will see something like this:

[![image.png](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/scaled-1680-/HCX5VSa7OOY5gn5s-image.png)](https://wiki.galaxydump.com/uploads/images/gallery/2025-09/HCX5VSa7OOY5gn5s-image.png)

If healthy, you will see one node as leader, and the others as voting followers.

<p class="callout info">NOTE: Make sure each node you configured, is present.</p>