HashiCorp Vault

Generate Certificates with Hashicorp Vault
Here are steps to generate SSL certificates using HashiCorp Vault as an Intermediate CA. 

 NOTE: Be sure that you've setup a vault instance as an Intermediate CA. See this page for how: Vault as Intermediate CA 

 Login to the web UI of your intermediate CA, such as:  https://vault02.ogsofttech.lan:8200/ui/ If DNS is down, use this: https://192.168.60.6:8200/ui/ 

 For the latest intermediate CA url, see this page: Vault Services 

 Find the issuing role by navigating to Secrets/PKI/Roles. 

 

 Select the role, and click Generate Certificate: 

 

 Fill in the Common name as: router.ogsofttech.lan. 

 Set the TTL to 1 year (365 days). 

 

 Click Generate, to create the key and certificate, and you’ll see this: 

 

 Download the private key as: router.ogsofttech.lan-key.pem 

 Download the certiticate as: router.ogsofttech.lan-cert.pem 

 Download the CA chain as: router.ogsofttech.lan-cabundle.pem 

 NOTE: We are calling the downloaded CA chain file a “ca bundle”. CA bundle is the standard naming convention for this file type. Specifically, a cert is often concatenated with the CA bundle that signed it, to create a chain certificate file. 

 Now, you can copy the cert, ca bundle, and private key to the host, for usage. 

 If generating a pair for a linux host, you will need them as .crt and .key files. Follow this: Converting PEM to crt and key 

 If generating a pair for an Nginx host, you will need

Hashicorp Vault Setup
Here are steps for setting up a secrets store using Hashicorp Vault, on Ubuntu 24. 

 References 

 Lots of steps were taken from here: https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engine 

 Server Setup 

 Before installing Vault, perform steps from this page, to setup the server: Ubuntu Host Setup 

 Since a typical vault cluster has limited internet visibility, it may be necessary to map in the local NTP server, to keep each node in sync. See this page for how to setup each host/VM to use the local private NTP server: Ubuntu: Use Private NTP Server 

 Install Vault 

 Here are three lines to install vault. 

 First, we will setup the package source keys… 

 sudo wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg 

 Now, add the package source entry… 

 echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list 

 Now, install vault… 

 sudo apt update && sudo apt install vault 

 Vault User 

 The installer created a user called, vault, and the service runs under it. But, the default keys are restricted to root permissions. So, we need to fix access to certificates. 

 For this, we will create a group to make it easier to control and test access to certs and keys. 

 sudo groupadd pki 

 And, we will add the vault user to the group: 

 sudo usermod -a -G pki vault 

 

 Clustered Config 

 NOTE: If you are standing up a single node vault, skip this section. 

 Work through the steps on this page, for clustering your vault servers: Clustering HashiCorp Vault 

 Vault Configuration 

 NOTE: This is for a single vault node. If you are standing up a cluster, skip this section. 

 Edit the config file at: /etc/vault.d/vault.hcl 

 Specifically, you want to set the tls cert/key paths to your own domain certs. 

 As well, you may want to limit the listener to just a single address, since the default is for all adapters on the host. 

 Once done, save the config update. 

 Continue Setup 

 Once installed and configured, we need to enable and start the vault services, with these lines: 

 sudo systemctl daemon-reload

sudo systemctl enable vault

sudo systemctl start vault

sudo systemctl status vault 

 You should be able to verify that the vault service is active by call this: 

 https://<host-ip>:8200/v1/sys/seal-status 

 NOTE: Use your VM’s fully-qualified name (or IP address), in the above. 

 If successful, the above call will return something like this: 

 {

 "type":"shamir",

 "initialized":false,

 "sealed":true,

 "t":0,

 "n":0,

 "progress":0,

 "nonce":"",

 "version":"1.17.6",

 "build_date":"2024-09-24T19:48:40Z",

 "migration":false,

 "recovery_seal":false,

 "storage_type":"file"

} 

 Cluster Unseal 

 If you're standing up a cluster, see this page: HashiCorp Vault Cluster Unseal 

 If you're standing up a single-node vault instance, see this page: Vault Single-Node Unseal 

 Administrative Setup 

 Once the vault is unsealed, you need to setup auditing and administrative policies. 

 See this page for how:  Vault Administrative Setup 

 Creating Users (Access Tokens) 

 Once the vault cluster is setup, you need to establish some administrative users.  

 See this page for how to create admin and user tokens: Vault Token Administration 

 To protect tokens in transit, see this page for Response-Wrapped Tokens:  Vault Wrapping Token 

 Login 

 Once your vault is unsealed, you can log into its UI. 

 Log into its UI, here:  http://<host-ip>:8200/ 

 You will be prompted to enter your root token, here: 

 

 If successful, you will see the root user’s dashboard: 

 

 Creating the First Secret Store 

 Now, you should create a secrets engine, to store things. The one that makes the most sense, first, is to make a KV store. You should be able to make a kv secrets store (actually, a kv-v2 store) from the dashboard: 

 

 The above will create the secrets store at the path: /kv You can change this if needed. 

 Once created, you will see the new empty store: 

 

 Using Vault as a CA 

 Follow steps on this page, to setup a vault instance as a Root CA: Vault as Root CA 

 NOTE: You never want to directly sign certificates with a Root CA. If you do, there is no way to revoke the root CA, without collapsing all PKI chains (by revoking the root CA certificate). However. You can create an Intermediate CA that does signings. And, that Intermediate CA can be revoked, without losing the root CA certificate. 

 So, we will create an Intermediate CA, to do actual signings. And, the Intermediate CA's certificate will be signed by the Root CA, just before offlining the Root CA. 

 Once a root CA is setup, you can follow this page, to setup an Intermediate CA, that will do actual signings: Vault as Intermediate CA 

 See this page for how to generate certificates: Generate Certificates with Hashicorp Vault

Clustering HashiCorp Vault
Here are special instructions for setting up a vault cluster. 

 NOTE: See the regular setup page for other details:  Hashicorp Vault Setup 

 DNS Resolution 

 Since the vault services will communicate with eachother over TLS, they will need certificates. And as such, the certs will include hostnames. 

 So, open the /etc/hosts file of each vault host, and add entries, at the bottom of the file, for each instance and API host. 

 Here's an example list of entries for a cluster: 

 192.168.75.10 vault02api

192.168.75.21 vault0201

192.168.75.22 vault0202

192.168.75.23 vault0203

192.168.75.24 vault0204

192.168.75.25 vault0205

192.168.75.26 vault0206 

 Filesystem Changes 

 Raft Folder 

 The Vault service will be running Raft. So, it will need a folder for the Raft backend. 

 NOTE: This may mean that the folder /opt/vault/data is obsolete. But, we will not worry about that, for now. The 'data' folder was created by the installer as the FS location for a storage = 'file' backend. 

 Create the raft folder with these: 

 sudo mkdir /opt/vault/raft 

 sudo chmod 700 /opt/vault/raft 

 sudo chown vault:vault /opt/vault/raft 

 TLS Folder 

 The installer already created the TLS folder, to store certificates. It is at: /opt/vault/tls 

 We will leave it as is. 

 Config Folder 

 The installer created a config folder at: /etc/vault.d 

 We need to bolster its permissions, as it may contain seal stanzas. 

 Update permissions of the config folder with these: 

 sudo chmod 0750 /etc/vault.d 

 sudo chown root:vault /etc/vault.d 

 Firewall Rules 

 Update the local firewall rules for each vault host, to allow 8200 and 8201 access. 

 sudo ufw allow 8200 

 sudo ufw allow 8201 

 Certificates 

 We will create certificates for each vault instance, and put them in the tls folder at: /opt/vault/tls 

 Follow instructions, here, to generate certificates for each host: Generate Certificates with Hashicorp Vault 

 NOTE: Be sure to do the following: - Set the common name to the fully qualified name: ex: vault0204.ogsofttech.lan. - Set the expiry to two years (17520 hours). - Set set the SAN IP to the address of the host: 192.168.75.24 

 Copy the CA certificate bundle (CA chain of issuer and root CA) into: /opt/vault/tls/ca.crt 

 NOTE: The ca.crt file should be the certificate bundle of issuer CA cert and root CA certificate. These should be bundled (root + intermediate) in the ca.crt file, as a concatenated PEM. 

 Concatenate the vault service certificate with the issuer CA cert as a concatenated PEM. 

 Paste the certificate file (vault service + issuer CA cert) into: /opt/vault/tls/vault.crt 

 NOTE: The vault.crt file should include the leaf certificate (of the node) plus the signing intermediate. 

 Copy the vault service private key into: /opt/vault/tls/vault.key 

 Once key and certs are stored, we need to set permissions on the files, with these: 

 # Ownership: keep vault:vault

sudo chown vault:vault /opt/vault/tls/vault.crt /opt/vault/tls/vault.key /opt/vault/tls/ca.crt

# Permissions:

# private key: only vault access

sudo chmod 0600 /opt/vault/tls/vault.key

# server cert usually fine as world-readable

sudo chmod 0644 /opt/vault/tls/vault.crt

# CA cert usually fine as world-readable

sudo chmod 0644 /opt/vault/tls/ca.crt 

 Here's a quick sanity check, to verify the certificates on a host: 

 # Verify the server chain file against your trust bundle

openssl verify -CAfile /opt/vault/tls/ca.crt /opt/vault/tls/vault.crt

# See what the gateway/clients will see

openssl s_client -connect vault0204:8200 -showcerts -verify_return_error \

 -CAfile /opt/vault/tls/ca.crt </dev/null 

 Vault.HCL Changes 

 Now, we need to create a config file for each vault service (vault.hcl). 

 There will be some tailoring required for each host. So, pay attention to the notes. 

 Open the vault config file at: /etc/vault.d/vault.hcl 

 sudo nano /etc/vault.d/vault.hcl 

 Below is a config file for a single node in a vault cluster. 

 NOTE: You will need to change the following lines for each host:     api_addr     cluster_addr     node_id 

 # Vault Configuration for Clustering.

# Created on 20250901

ui = true

cluster_name = "vaultcluster2"

# recommended with Integrated Storage

disable_mlock = true

# Advertise addresses (MUST be correct & reachable)

# Define the external API address that clients will use to communicate with this Vault node.

# set per node

api_addr = "https://vault0204.ogsofttech.lan:8200"

# Define the internal address used by Vault nodes to communicate with this Vault node.

# Vault ignores the scheme of this URL, so it doesn't matter if http or https.

# set per node (node's intra-cluster IP:8201)

cluster_addr = "https://192.168.75.24:8201"

listener "tcp" {

 address = "0.0.0.0:8200"

 tls_disable = 0

 tls_cert_file = "/opt/vault/tls/vault.crt"

 tls_key_file = "/opt/vault/tls/vault.key"

 tls_client_ca_file = "/opt/vault/tls/ca.crt"

 # optional: require client certs from automation:

 # tls_require_and_verify_client_cert = "true"

}

# Updated to use raft backend (Integrated Storage).

storage "raft" {

 path = "/opt/vault/raft"

# unique on each node 

 node_id = "vault0204"

 # Auto-join peers (preferred over manual joins)

 retry_join {

 leader_api_addr = "https://vault0204.ogsofttech.lan:8200"

 leader_ca_cert_file = "/opt/vault/tls/ca.crt"

 }

 retry_join {

 leader_api_addr = "https://vault0205.ogsofttech.lan:8200"

 leader_ca_cert_file = "/opt/vault/tls/ca.crt"

 }

 retry_join {

 leader_api_addr = "https://vault0206.ogsofttech.lan:8200"

 leader_ca_cert_file = "/opt/vault/tls/ca.crt"

 }

}

# Recommended health endpoint behavior for LBs

# (defaults are fine; LB should treat 200 as active, 429 as standby, 503 sealed)

# See /v1/sys/health docs for details.

 

 Service File (Systemd Unit) 

 With folders fixed, certs and config defined, we need to configure the vault service for operation. 

 To do so, we need to identify the service file that the installer created, with this: 

 sudo systemctl cat vault.service 

 As of 20250803, the installer puts the systemd unit file, here: 

 /usr/lib/systemd/system/vault.service 

 We need to open it and update it: 

 sudo nano /usr/lib/systemd/system/vault.service 

 Once updated, save and close the systemd file. 

 Return to the generic vault setup page: https://wiki.galaxydump.com/link/434#bkmrk-continue-setup

HashiCorp Vault Cluster Unseal
These steps are for a new vault cluster that has been configured and started up, but is in an unsealed state. 

 For an existing cluster with unsealed nodes, see this page: Handling Vault Node Restart 

 Leader Initialization 

 The following will initialize a new vault cluster and return a set of unseal keys and a root token. 

 NOTE: The ca.crt file is privileged, You will need to run these commands as the root user. Run the following to switch to the root user: 

 sudo -i 

 Go to the first node, and do these (as root): 

 # From an admin shell that can reach the VLAN:

export VAULT_ADDR="https://vault0204.ogsofttech.lan:8200"

export VAULT_CACERT="/opt/vault/tls/ca.crt" # path on your admin box

# Initialize the cluster (choose your own shares/threshold)

vault operator init -key-shares=5 -key-threshold=3 

 NOTE: Use the fully-qualified hostname above, as it appears in the node's cert. 

 Once executed, the vault node will reply with 5 unseal keys and an initial root token. 

 Distribute each of these unseal keys to trusted admins, to store in offline password storage. 

 NOTE: Three (3) unseal keys are required to unseal the vault. 

 Use the initial root token to setup policies and auth. Then, retire it. 

 Unseal the Leader 

 With the unseal keys from the initialized node (received above), we need to unseal its vault. 

 NOTE: We do this, while still as root, and on the same host that we got the keys from. 

 Now, unseal each node, by calling this command once each, for three of the five unseal keys: 

 NOTE: It will prompt you for the unseal key, each time you run it. 

 vault operator unseal 

 Initial Root Login 

 With the vault unsealed, we need to perform an initial login as root: 

 # Log in with the root token for initial setup tasks

vault login <root_token> 

 Once logged in, you can check the vault status with this: 

 vault status 

 If successful, you should see Initialized: true , Sealed: false , HA Enabled: true , and this node as leader. 

 The first node is online, and the cluster is up... sort of. 

 Each cluster member has auto-discovered a leader and established a RAFT quorum. 

 But, the other nodes are still not unsealed (since we did not configured auto-unseal). 

 Unseal Other Nodes 

 Similar to what you did, to unseal the first node, we will do the same to each member, below. 

 Switch to root on each node with: 

 sudo -i 

 Set exports for each node: 

 NOTE: Make sure that the vault_addr variable is pointing to the local node being unsealed, here. 

 export VAULT_ADDR="https://vault0205.ogsofttech.lan:8200";

export VAULT_CACERT="/opt/vault/tls/ca.crt" 

 NOTE: Use the fully-qualified hostname above, as it appears in the node's cert. 

 Now, unseal each node, by calling this command once each, for three of the five unseal keys: 

 NOTE: It will prompt you for the unseal key, each time you run it. 

 vault operator unseal 

 Check Status 

 Once you have initialized the leader node, and unsealed all nodes, we need to confirm that the cluster is good. 

 Run this: 

 NOTE: The vault_addr should be pointing to the leader, here. 

 # Set this if you are coming back to this page, and the environment value is not set...

export VAULT_ADDR="https://vault0204.ogsofttech.lan:8200";

# Run this to check status...

vault status 

 Confirm RAFT peers with this: 

 vault operator raft list-peers 

 NOTE: The above may only work on the current leader, because https. We need to work through why this is, and solve it, so it can be run on any node. 

 When run, you will see something like this: 

 

 If healthy, you will see one node as leader, and the others as voting followers. 

 NOTE: Make sure each node you configured, is present. 

 There is a health check that can be performed, by calling this: 

 curl -s -H "X-Vault-Token: $VAULT_TOKEN" "$VAULT_ADDR/v1/sys/ha-status" | jq . 

 NOTE: Be sure that the VAULT_TOKEN and VAULT_ADDR environment variables are set. Or, you can hardcode them with a minimal privilege user account. 

 When run, you will get a JSON list of nodes and their status.

Vault Wrapping Tokens
When creating access tokens for HashiCorp Vault, you always want to prevent them from falling into the wrong hands, or showing up as clear-text in command line history, logs and audit trains. 

 To reduce the chance of tokens being passed in the clear, you can create a new user token in a response-wrapped state. This allows the token to be given to a user, over chat, or text, without much concern. 

 The response-wrapped token has a very short lifetime, and can only be redeemed once. 

 Here's how to create a response-wrapped token: 

 vault token create -role=admin -orphan -wrap-ttl=5m -format=json 

 The command response will include the wrapping token, like this: 

 {

 "request_id": "",

 "lease_id": "",

 "lease_duration": 0,

 "renewable": false,

 "data": null,

 "warnings": null,

 "wrap_info": {

 "token": "hvs.tH5Wn8bD3eJxvMq1iP7F",

 "accessor": "PNowpcQP0jJ9Jpz06o2oueYW",

 "ttl": 300,

 "creation_time": "2025-09-04T02:26:49.859843035Z",

 "creation_path": "auth/token/create/admin",

 "wrapped_accessor": "5KlfcQwbo05Ej37YOqBJnfHM"

 }

} 

 The wrap_info/token property is what you give to the user. 

 The user can then, redeem their access token by submitting the wrapping token, like this: 

 vault unwrap hvs.tH5Wn8bD3eJxvMq1iP7F 

 Vault will respond with the real token, and revoke further usage of the wrapping token. 

 NOTE: The wrapping token has an expiry. If exceeded, the user will need to request another. 

 Wrapped Token Benefits 

 Using wrapped tokens, prevents exposure in shell history or logs. 

 Delivery is safer, as you can drop the short-lived wrapper into a config management pipeline or paste in a chat. 

 The Vault audit log records that the wrapping token was created and who unwrapped it.

Vault Token Administration
Here are notes on access token administration. 

 Be sure that you've already setup an administrative policy in your vault cluster. See this page for how: Vault Administrative Setup 

 Creating Admin Tokens 

 Once the admin policy exists, you can create administrative access tokens. 

 To do so, log into the leader note, with this: 

 # Use the root token...

vault login <root_token> 

 You can create a simple admin token that lasts 24 hours: 

 # Create an admin token that lasts 24 hours...

vault token create -policy=admin -ttl=24h 

 The above will issue short-lived admin tokens, so that the vault is more protected. These can be minted by an automated script, that issues token as needed. 

 Traditional Admin 

 A more traditional administrative token needs to be longer-lived than a 24 hour token. 

 So, we will create a token role for admins. 

 This will create a periodic, orphan role so that admin tokens can be renewed forever (until revoked), and aren't tied to a parent token: 

 NOTE: You will need to be logged into the vault leader (vault login <token>). 

 vault write auth/token/roles/admin \

 allowed_policies="admin,default" \

 orphan=true \

 renewable=true \

 period=720h 

 The above token role, can create tokens that last 30 days, and can be renewed, without changing the token. 

 With the above admin role, you can create admin tokens of a more traditional lifetime, with this: 

 vault token create -role=admin -orphan -format=json 

 The above command will create admin tokens that can be given to your ops group, for longer-lived administrative access. 

 NOTE: Be sure to save the client_token field in a secure place, such as a private password manager. 

 Token Renewal 

 Before a token expires, you can renew it with this command: 

 VAULT_TOKEN=<admin_token> vault token renew 

 NOTE: The above can be automated, if the admin token is used by scripts or services. (Automate with a small systemd timer/cron on a secure host or in your gateway.) 

 Creating Other Admins 

 You can use an admin token to create other admins, like this: 

 # Another admin token (same role)

vault login <admin_token>

vault token create -role=admin -orphan -format=json 

 Creating Tokens for other Groups 

 Once you have other roles established as policies, you can create tokens for users, in those roles/policies, like this: 

 vault token create -policy=kv-readers -ttl=8h 

 Response-Wrapped Tokens 

 See this page for Response Wrapped Tokens: Vault Wrapping Token

Vault Single-Node Unseal
Here are instructions on how to unseal a single-node vault cluster. 

 Vault CLI Comms 

 For the vault command to communicate with the running service, we must set an env variable for it: 

 export VAULT_ADDR=https://your_domain:8200 

 NOTE: You may have to set the above to http, if you haven’t given vault a valid ssl cert, yet. 

 Initialize the Vault 

 In order for the vault service to manage secrets, you must initialize its store. To do this, you need to run the following: 

 vault operator init -key-shares=3 -key-threshold=2 

 You can change the key shares and threshold based on your risk profile. 

 The vault CLI will respond with something like this: 

 Unseal Key 1: eZcJeydRrqeSMZ1zTN+VVll9TFT2KvJy7VlnxUgtvuz5

Unseal Key 2: ntmqCKq8rgNnKT1YSLCjVmCCZBAA3NwUeqxIyRpYD4Wm

Unseal Key 3: 3FK1+Hsorh4p8/L9mki3VskaEU2eQhLqGOI/pJkTHMbx

Initial Root Token: s.hY0ieybfDqCadz7JpL88uO3x 

 Save these values in a secure location, as this is the only time you will receive them. 

 Now, you can run the vault status command, and see it has changed to Initialized=true. 

 Run this to get vault status: 

 vault status 

 

 Now, you need to unseal the vault, so it can be used. 

 Notice the vault status showed an unseal progress of 0/2. This means, that at least two more unseal tokens must be submitted for it to be available for access. 

 Enter enough tokens to unlock your vault with this command, and paste in one of the tokens when prompted: 

 vault operator unseal 

 Once the unseal threshold has been met, the Sealed state will become ‘false’. 

 

 Your vault is available for access.

Vault Administrative Setup
Once you have a vault instance or cluster unsealed, you can setup auditing, and administrative policies, with these instructions. 

 See this page for how to setup a vault instance or cluster: Hashicorp Vault Setup 

 Audit Logging 

 Create a folder for capturing audit logs on each node (leader and followers): 

 sudo mkdir -p /var/log/vault

sudo chown vault:vault /var/log/vault

sudo chmod 0750 /var/log/vault 

 Tell the leader node to store audit logs in the created folder: 

 vault audit enable file file_path=/var/log/vault/audit.log 

 Admin Policy 

 WARNING: The initial root token has broad privilege, and bypasses all policy checks, being meant for bootstrapping and emergencies. You need to create an actual admin policy, so that proper security and auditing can occur.  

 Here, we will setup an admin policy, so that we can stop using the initial root token. 

 Doing so, gives us many benefits: 

 

 Users created under the admin policy share the same privileges as the root user. 

 We can issue short-lived admin tokens. 

 We can audit all admin actions (audit logs will show policy=admin and the user, not just root). 

 We can later, reduced privileges, instead of everyone having anonymous root access. 

 

 NOTE: We only have to create the admin policy once. We can do it on the current leader. 

 Create a file called: admin.hcl: 

 sudo nano /etc/vault.d/admin.hcl 

 NOTE: It doesn't matter where we generate the admin.hcl file, as the policy write command will pull it into the vault. But for simplicity, we will create it on the leader node, so we have easy tracking of what setup steps have been done. 

 Give the admin policy file, this content: 

 # --- token management (mint/lookup/renew/revoke/roles) ---

path "auth/token/create" { capabilities = ["update"] }

path "auth/token/create/*" { capabilities = ["update"] }

path "auth/token/lookup" { capabilities = ["read", "update"] }

path "auth/token/lookup-self" { capabilities = ["read"] }

path "auth/token/renew" { capabilities = ["update"] }

path "auth/token/renew-self" { capabilities = ["update"] }

path "auth/token/revoke" { capabilities = ["update"] }

path "auth/token/revoke-self" { capabilities = ["update"] }

path "auth/token/roles" { capabilities = ["list", "read"] }

path "auth/token/roles/*" { capabilities = ["create","read","update","delete","list"] }

# --- policies (so admins can maintain policies without root) ---

path "sys/policies/acl" { capabilities = ["list"] }

path "sys/policies/acl/*" { capabilities = ["create","read","update","delete","list"] }

# --- core admin knobs (optional but typical) ---

path "sys/audit" { capabilities = ["read"] }

path "sys/audit/*" { capabilities = ["create","read","update","delete","sudo"] }

path "sys/auth" { capabilities = ["read"] }

path "sys/auth/*" { capabilities = ["create","read","update","delete","sudo"] }

path "sys/mounts" { capabilities = ["read"] }

path "sys/mounts/*" { capabilities = ["create","read","update","delete","sudo"] }

# --- give admin wide access to secrets during bootstrap (tighten later) ---

path "*" { capabilities = ["create","read","update","delete","list","sudo"] }

 

 Once created, save and close. 

 Here are some notes about the above policy: 

 

 The top block allows an admin to perform all token management actions. 

 The middle block allows admins to manipulate policies. 

 The third block allows admins to perform other root actions. 

 The bottom block (path "*") allows access to all paths in the vault (like root).

 

 capabilities = create/read/update/delete/list - These allow full browsing and editing. 

 capabilities = sudo - allows doing system-level actions like:

 

 enabling audit devices, 

 enabling auth methods, 

 mounting new secrets engines, 

 etc. 

 

 

 

 

 

 So, the above admin policy is just like root, but allows us to have named users as admins. 

 Enable the admin policy with this: 

 vault policy write admin admin.hcl 

 Once written, the policy is stored in RAFT, and replicated to all nodes. 

 We can confirm the policy with this: 

 vault policy read admin 

 Now, any tokens with the admin policy, will function with the above administrative privileges. 

 Creating Admin Tokens 

 Once the admin policy exists, you can create administrative access tokens. 

 See this page for how to create admin and user tokens: Vault Token Administration 

 To protect tokens in transit, see this page for Response-Wrapped Tokens: Vault Wrapping Token

Vault as Root CA
Here are steps you can follow to setup a vault instance as a Root CA. 

 NOTE: This page assumes that you have created a single-node vault instance to serve as your Root CA. See this page for how to do that: Hashicorp Vault Setup 

 NOTE: These steps will create a root CA with one signing key. You should create an intermediate CA, as well, that will perform the actual signing of certs. This will allow you to offline this root CA, once the intermediate CA is up. 

 Starting the PKI Engine 

 NOTE: From here down, is steps for creating a Root CA. If you are setting up an Intermediate CA, skip to ‘Configure Intermediate CA’. 

 For Vault to serve as a root CA, you have to add the PKI secrets engine. 

 To do this, enable the pki secrets engine: 

 

 For a Root CA, set the Max lease time to as long as possible, as the root CA will be offlined after generation. 

 Once initialized, it will look like this: 

 

 For a root CA, click Configure PKI to begin setup. 

 

 Choose the Generate Root option. 

 Set Type to Internal. 

 Set your Common name. Usually, this is a domain with a private TLD suffix. 

 Set the issuer name. 

 Set a TTL that is maxed out: 87600 hours. 

 

 Scroll down and fill in the issuer URLs, matching the origin for your root CA host: 

 

 When finished, click Done, to generate the root CA key and certificate. 

 You will see a page like this: 

 

 Copy out the root CA certificate, and save it to a file, named: ogsofttech.lan_ca.crt 

 Install it on all machines of the local network, so they will recognized the signed SSL certs of hosts. 

 Current Intranet Root CA Certificate 

 The current root CA for the local intranet can be found in the secure share at this path:  

 \SecureShare git\oga\ogsofttech.lan\rootCA 

 NOTE: This certificate should be installed on any host that will consume services signed by the root ca. 

 See this page for how to install it on an Ubuntu host: How to Add Root CAs to Ubuntu 

 Root CA Rotation 

 To make things easier, when it comes time to rotate your root CA key, add a role, now. 

 Click PKI. 

 Click Roles. 

 Click Create Role. 

 Give the new role a name that you will recognize as the root CA rotation role: CA_rotation_role 

 Leave the rest of the fields empty, and click Create. 

 Now, you have a working root CA key/cert for your network. 

 We will use it to sign the certificate of an Intermediate CA, that will do all the actual work. 

 And, we will offline the root CA.

Vault as Intermediate CA
Here are instructions for setting up a vault instance as an Intermediate CA. 

 NOTE: This page assumes that you have created a single-node vault instance to serve as your Root CA. See this page for how to do that: Hashicorp Vault Setup 

 NOTE: Always create the Intermediate CA as a separate instance from your Root CA. This allows you to offline the Root CA, to uphold the integrity of your root CA certificate. 

 To properly sign certificates, you will want to generate an Intermediate CA that will do all the actual issuing. And, you will offline the root CA, after the Intermediate CA is usable. 

 See this page if you need to setup a vault instance as a Root CA: Vault as Root CA 

 Configure Intermediate CA 

 For Vault to serve as an Intermediate CA, you have to add the PKI secrets engine. 

 To do this, enable the pki secrets engine: 

 

 For an Intermediate CA, set the Max lease time to 43800 hours, and enable the engine. 

 Once initialized, it will look like this: 

 

 For an Intermediate CA, click Configure PKI to begin setup. 

 

 Choose the Generate intermediate CSR option, and fill in the blanks. 

 Set Type to Internal. 

 Set your Common name to your domain with a suffix that it’s an Intermediate CA, like this: 

 ogsofttech.lan Int CA 01 

 Click Generate, to make the CSR. 

 

 Paste the CSR onto the clipboard, and save it as a file: pki_intermediate.csr . 

 Open the web UI for your root CA, and navigate to Secrets/PKI/Issuers, and click Sign Intermediate. 

 

 Paste the CSR content into the CSR field. 

 Set the domain name as the common name: ogsofttech.lan 

 Select pem_bundle from the Format dropdown: 

 

 Click Sign, to sign the CSR from the Intermediate CA. 

 Once signed, you’ll see this: 

 

 It contains the signed certificate for the Intermediate CA, the cert of the issuing CA, and the CA chain cert. 

 NOTE: Save all three in a known place, as this is the only time these will be available. 

 Download the certificate as: ogsofttech.lan-intca01.cert.pem 

 Open the UI for the Intermediate CA, and navigate to the Import a CA: 

 NOTE: You get to this, by clicking on the Issuers tab, and clicking Import. 

 

 Upload the certificate that you saved to: ogsofttech.lan-intca01.cert.pem 

 Click Import Issuer. 

 The Issuers tab will show two entries, one for the root, and one for the Int CA: 

 

 Issuing Role 

 Before you can issue certificates to hosts and such, you need to create an issuer role. 

 To do this, we need to identify the issuer’s Guid. We can get this by clicking the terminal button, and running this: 

 read -field=default pki/config/issuers 

 Note the Guid that was returned. 

 

 Now, go to Secrets/PKI/Roles, and create a new role. 

 Give the role a name in this format: ogsofttech-dot-lan 

 Unclick the Use Default Issuer, and select the Guid that matches what we found above. 

 Set the TTL to 43800 hours. 

 

 At the bottom of the form… 

 In the Allowed Domains field, add our domain: ogsofttech.lan. 

 Check the “Allow Subdomains” checkbox. 

 

 And, click Create, to make our issuing role. 

 

 Now, you have a role that can create and sign certificates. 

 If not already done, you can offline your root CA, as it needs to be kept safe, and only needed to create a new Intermediate CA. 

 See this page for how to generate certificates: Generate Certificates with Hashicorp Vault

Handling Vault Node Restart
Each time a Vault node restarts, you will have to unseal it, so that it can participate in the cluster. Here are steps to do that. 

 Switch to root on the node with: 

 sudo -i 

 Set exports for the node: 

 NOTE: Make sure that the vault_addr variable is pointing to the local node being unsealed, here. 

 export VAULT_ADDR="https://vault0205.ogsofttech.lan:8200";

export VAULT_CACERT="/opt/vault/tls/ca.crt" 

 NOTE: Use the fully-qualified hostname above, as it appears in the node's cert. 

 Now, unseal each node, by calling this command once each, for three of the five unseal keys: 

 NOTE: It will prompt you for the unseal key, each time you run it. 

 vault operator unseal 

 Once unsealed, you can verify cluster membership with this: 

 vault operator raft list-peers 

 When run, you will see something like this: 

 

 If healthy, you will see one node as leader, and the others as voting followers. 

 NOTE: Make sure each node you configured, is present.