Nomad TLS Configuration: Securing Cluster Communication

Video

The video above walks through the full setup live. Below is the extended write-up with every command, error explanation, and configuration file.

Overview

HashiCorp Nomad uses mutual TLS (mTLS) to authenticate communication between:

Server ↔ Server
Server ↔ Client
CLI ↔ Server (via the API)

Without TLS, any node can join your cluster and any process can hit the API. This guide sets it up from scratch using Nomad’s built-in CA tooling.

Prerequisites

Nomad ≥ 1.5 installed on all nodes
A user with sudo on each node
Ports 4646 (HTTP), 4647 (RPC), 4648 (Serf) open internally

Step 1 — Create the Certificate Authority

Nomad ships with nomad tls ca create so you don’t need external tools.

# Run this once on a secure machine (not a Nomad node)
nomad tls ca create

This produces two files:

nomad-agent-ca.pem — the CA certificate (distribute to all nodes)
nomad-agent-ca-key.pem — keep this secret, never put it on a Nomad node

Step 2 — Generate Server Certificates

# Generate certs for the server in region "global"
nomad tls cert create -server -region global -dc dc1

# Output:
# ==> WARNING: Server Certificates grants authority to become a
#     temporary Nomad server node. Treat it as you would a root certificate.
# --> Wrote nomad-server-global-0.pem
# --> Wrote nomad-server-global-0-key.pem

Repeat for each additional server (increment the number).

Step 3 — Generate Client Certificates

nomad tls cert create -client -region global -dc dc1
# --> Wrote nomad-client-global-0.pem
# --> Wrote nomad-client-global-0-key.pem

Step 4 — Distribute Certificates

# Copy CA cert to all nodes
scp nomad-agent-ca.pem user@server1:/etc/nomad.d/
scp nomad-agent-ca.pem user@client1:/etc/nomad.d/

# Copy server certs to server nodes only
scp nomad-server-global-0.pem nomad-server-global-0-key.pem user@server1:/etc/nomad.d/

# Copy client certs to client nodes
scp nomad-client-global-0.pem nomad-client-global-0-key.pem user@client1:/etc/nomad.d/

# Lock down permissions
chmod 640 /etc/nomad.d/*.pem
chown root:nomad /etc/nomad.d/*.pem

Step 5 — Configure TLS in nomad.hcl

Server configuration (/etc/nomad.d/nomad.hcl on server nodes):

tls {
  http = true
  rpc  = true

  ca_file   = "/etc/nomad.d/nomad-agent-ca.pem"
  cert_file = "/etc/nomad.d/nomad-server-global-0.pem"
  key_file  = "/etc/nomad.d/nomad-server-global-0-key.pem"

  verify_server_hostname = true
  verify_https_client    = true
}

Client configuration:

tls {
  http = true
  rpc  = true

  ca_file   = "/etc/nomad.d/nomad-agent-ca.pem"
  cert_file = "/etc/nomad.d/nomad-client-global-0.pem"
  key_file  = "/etc/nomad.d/nomad-client-global-0-key.pem"

  verify_server_hostname = true
  verify_https_client    = true
}

Step 6 — Configure the CLI

export NOMAD_ADDR=https://server1:4646
export NOMAD_CACERT=/etc/nomad.d/nomad-agent-ca.pem
export NOMAD_CLIENT_CERT=/etc/nomad.d/nomad-server-global-0.pem
export NOMAD_CLIENT_KEY=/etc/nomad.d/nomad-server-global-0-key.pem

nomad node status

Add these exports to ~/.bashrc or /etc/profile.d/nomad.sh.

Common Errors

`x509: certificate signed by unknown authority`

The CA cert isn’t trusted. Verify:

openssl verify -CAfile nomad-agent-ca.pem nomad-server-global-0.pem
# Should print: nomad-server-global-0.pem: OK

If it fails, you generated the cert with a different CA. Regenerate.

`x509: certificate is valid for server.global.nomad, not ...`

The hostname in the cert doesn’t match. Check your -region and -dc flags match your Nomad config.

`connection refused` after enabling TLS

Make sure you updated NOMAD_ADDR from http:// to https://.

💡 Additional Tips

Rotate certificates before expiry: nomad tls cert create generates 1-year certs by default. Set a reminder.
Use verify_https_client = false temporarily during initial setup if you’re locked out — but never in production.
Store your CA key in a vault (HashiCorp Vault, offline USB, or a secrets manager). Compromise of the CA key means all certs are compromised.