How to rotate to a new certificate authority
Picture this: you're looking at your Grafana dashboard (using stats from Nebula's Prometheus endpoint) when you notice your certificate authority (CA) is expiring in a month. When it expires your host certificates will no longer be valid and your hosts will stop communicating with each other. It's time to rotate your CA and host certificates!
In this guide, we'll walk through the process of creating a new certificate authority, updating the Nebula CA trust bundle on your existing nodes, and minting fresh host certificates signed by the new CA. Once all of your hosts have been updated with certificates from the new CA, we'll remove the old CA from the trust bundle.
Considerations
When thinking about CA rotation, it's best to plan in advance. Since you'll need access to the config file on every host in your network, it's a good idea to start rotating certificates early, possibly even months in advance, to avoid any connection issues. Ideally your hosts are managed using a config management tool like Ansible, Chef, or Puppet. If not, you'll need to manually connect to each device to update its Nebula config.
Additionally, consider checking your config and enabling
pki.disconnect_invalid
if you have not already. This flag will cause Nebula
to close existing tunnels to hosts which are no longer trusted - either because their certificates are expired or are no
longer signed by a trusted CA. This will accelerate how quickly you learn of problems after updating your CA trust
bundle, giving you as much time as possible to rollback and fix any issues before your CA expires.
Let's get started!
Step 1: Generate a new Certificate Authority
The first thing we need to do is create a new certificate authority with an expiration in the future. The new CA should
use the exact same CIDR, group, and subnet restrictions as the original certificate. You can use nebula-cert print
to
inspect your old certificate.
$ nebula-cert print -path ca.crt
NebulaCertificate {
Details {
Name: test ca - do not use
Ips: [
192.168.100.0/24
]
Subnets: []
Groups: []
Not before: 2022-07-31 16:08:16 -0400 EDT
Not After: 2023-07-30 16:08:16 -0400 EDT
Is CA: true
Issuer:
Public key: 2976767da3dc58eb47cfe733e7daf4531fa9cd2ee5a320e548c65487a251de1a
Curve: CURVE25519
}
Fingerprint: d5978d6d54a58e4685551708c5f57fbdd3774be67d470ecb0033cf70bbf5fbb5
Signature: 5fab5ddb6b175274fe750013ec9a7a306ee4334f7a563c9c31a799a453618802aa2752d21f44720d876027d08b05d8a9da7dd61089eebf1a184773baf681de06
}
To match this certificate, we would run nebula-cert ca -name "test ca - do not use #2" -ips "192.168.100.0/24"
. We do
not need to pass -groups
or -subnets
because this CA has no such restrictions. By default, Nebula will set the
expiration to a year from today. If you'd like to use a custom expiration, you can use the -duration
flag.
Nebula offers built-in encryption of the CA private key since v1.7.0. If you do not plan to store the private key in encrypted storage (e.g. Ansible Vault or AWS Secrets Manager), it is recommended that you use the built-in encryption.
To encrypt your Nebula private key, pass the -encrypt
flag when generating the CA and you will be prompted for a
passphrase. Keep it safe - you will be prompted for it each time you sign a host using the encrypted CA key.
Before proceeding, we strongly recommend you set a reminder for yourself to rotate your CA again in the future. Consider setting a reminder in your team's shared calendar for 2-3 months prior to your new CA's expiration.
Step 2: Updating your existing host's trust bundle
Before issuing new certificates for each host, it's important to update the trust bundle in all of your existing hosts' configs. This will ensure that as hosts are moved over to certificates signed by the new CA, these new certificates will be trusted by the rest of your network.
On each host, find the pki
section and append the new CA underneath your existing CA in the ca
setting. For example,
if your CA is is inlined, it may look something like this:
pki:
ca: |
-----BEGIN NEBULA CERTIFICATE-----
... existing PEM-encoded CA certificate ...
-----END NEBULA CERTIFICATE-----
-----BEGIN NEBULA CERTIFICATE-----
... new PEM-encoded CA certificate ...
-----END NEBULA CERTIFICATE-----
cert: /etc/nebula/device.crt
key: /etc/nebula/device.key
If your CA trust bundle exists in a file (e.g. /etc/nebula/ca.crt
), the same instructions apply: simply append the new
PEM certificate below the existing certificate(s).
In order for this change to take effect, you must restart or reload Nebula. If you're using systemd,
systemctl restart nebula
will restart the process. Alternatively, to gracefully reload Nebula without tearing down
tunnels, find its pid in your process list and run kill -SIGHUP <pid>
. You can verify the config was reloaded by
looking for the following lines:
time="2022-07-27T11:32:04-04:00" level=info msg="Caught HUP, reloading config"
...
time="2022-07-27T11:32:04-04:00" level=info msg="Trusted CA certificates refreshed" fingerprints=[d5978d6d54a58e4685551708c5f57fbdd3774be67d470ecb0033cf70bbf5fbb5 b680813a506933b2020f6e9980fc26f7df79c8124302e32aedba23c820e813dc]"
For extra credit, you can verify that your newly minted CA's fingerprint appears in the log line shown above.
Step 3: Signing new host certificates
Now that all of your hosts have been updated to trust the new CA, it's time to issue new host certificates. You'll need to iterate all existing certificates and issue a new certificate using the same information as before:
$ nebula-cert print -path host1.crt
NebulaCertificate {
Details {
Name: host1
Ips: [
192.168.100.5/24
]
Subnets: [
10.0.0.0/8
]
Groups: [
"prod"
"api"
]
Not before: 2023-02-30 16:22:00 -0400 EDT
Not After: 2023-07-30 16:08:16 -0400 EDT
Is CA: false
Issuer: d5978d6d54a58e4685551708c5f57fbdd3774be67d470ecb0033cf70bbf5fbb5
Public key: 4a915591ff1a6869acb085d0292cbd25ba88624b9729420acb20d03644e0b016
Curve: CURVE25519
}
Fingerprint: 92efefd0575f71c10973dc96d9e2111d62703139383855f5a6a74feea68af05e
Signature: dc680011a11078fc00cce84d176662f54c96fa071d1bd49d5410a987f5743c3a641e27142ec19d5ed1929d5464bcdffe927a787b3a4f200b008d84821e3c4a0d
}
$ nebula-cert sign -name "host1" -subnets "10.0.0.0/8" -ip "192.168.100.5/24" -groups "prod,api"
This will create a new host1.crt
and host1.key
. Copy the new certificate and key to your host, update the Nebula
config (pki.cert
and pki.key
), and restart or reload Nebula
as you did in the previous step.
Even though this host is using a different CA than the rest of your network you should still be able to communicate with
the rest of your hosts because its certificate is signed by a CA in their trust bundle. You can issue a ping
to test
this now. Don't forget that your firewall must allow pings from this host.
Step 4: Removing the old CA from the trust bundle
You did it! All of your hosts have updated certificates and are communicating with each other. At this point, you can safely remove the original CA from the trust bundle so that only the new CA exists:
pki:
ca: |
-----BEGIN NEBULA CERTIFICATE-----
... new PEM-encoded CA certificate ...
-----END NEBULA CERTIFICATE-----
cert: /etc/nebula/device.crt
key: /etc/nebula/device.key
Congratulations on rotating your certificate authority. Don't forget to set a reminder before the next expiration!
Sick of manually managing IP addresses, certificates, and certificate authority expirations?
Defined Networking can help! We offer a Managed Nebula solution that removes the toil of managing your network. Try it today!