Skip to main content

How to rotate to a new certificate authority

Picture this: you're looking at your Grafana dashboard (using stats from Nebula's Prometheus endpoint) when you notice your certificate authority (CA) is expiring in a month. When it expires your host certificates will no longer be valid and your hosts will stop communicating with each other. It's time to rotate your CA and host certificates!

In this guide, we'll walk through the process of creating a new certificate authority, updating the Nebula CA trust bundle on your existing nodes, and minting fresh host certificates signed by the new CA. Once all of your hosts have been updated with certificates from the new CA, we'll remove the old CA from the trust bundle.

Considerations​

When thinking about CA rotation, it's best to plan in advance. Since you'll need access to the config file on every host in your network, it's a good idea to start rotating certificates early, possibly even months in advance, to avoid any connection issues. Ideally your hosts are managed using a config management tool like Ansible, Chef, or Puppet. If not, you'll need to manually connect to each device to update its Nebula config.

Additionally, consider checking your config and enabling pki.disconnect_invalid if you have not already. This flag will cause Nebula to close existing tunnels to hosts which are no longer trusted - either because their certificates are expired or are no longer signed by a trusted CA. This will accelerate how quickly you learn of problems after updating your CA trust bundle, giving you as much time as possible to rollback and fix any issues before your CA expires.

Let's get started!​

Step 1: Generate a new Certificate Authority​

The first thing we need to do is create a new certificate authority with an expiration in the future. The new CA should use the exact same CIDR, group, and subnet restrictions as the original certificate. You can use nebula-cert print to inspect your old certificate.

$ nebula-cert print -path ca.crt
NebulaCertificate {
	Details {
		Name: test ca - do not use
		Ips: [
			192.168.100.0/24
		]
		Subnets: []
		Groups: []
		Not before: 2022-07-31 16:08:16 -0400 EDT
		Not After: 2023-07-30 16:08:16 -0400 EDT
		Is CA: true
		Issuer:
		Public key: 2976767da3dc58eb47cfe733e7daf4531fa9cd2ee5a320e548c65487a251de1a
		Curve: CURVE25519
	}
	Fingerprint: d5978d6d54a58e4685551708c5f57fbdd3774be67d470ecb0033cf70bbf5fbb5
	Signature: 5fab5ddb6b175274fe750013ec9a7a306ee4334f7a563c9c31a799a453618802aa2752d21f44720d876027d08b05d8a9da7dd61089eebf1a184773baf681de06
}

To match this certificate, we would run nebula-cert ca -name "test ca - do not use #2" -ips "192.168.100.0/24". We do not need to pass -groups or -subnets because this CA has no such restrictions. By default, Nebula will set the expiration to a year from today. If you'd like to use a custom expiration, you can use the -duration flag.

note

Nebula offers built-in encryption of the CA private key since v1.7.0. If you do not plan to store the private key in encrypted storage (e.g. Ansible Vault or AWS Secrets Manager), it is recommended that you use the built-in encryption.

To encrypt your Nebula private key, pass the -encrypt flag when generating the CA and you will be prompted for a passphrase. Keep it safe - you will be prompted for it each time you sign a host using the encrypted CA key.

Before proceeding, we strongly recommend you set a reminder for yourself to rotate your CA again in the future. Consider setting a reminder in your team's shared calendar for 2-3 months prior to your new CA's expiration.

Step 2: Updating your existing host's trust bundle​

Before issuing new certificates for each host, it's important to update the trust bundle in all of your existing hosts' configs. This will ensure that as hosts are moved over to certificates signed by the new CA, these new certificates will be trusted by the rest of your network.

On each host, find the pki section and append the new CA underneath your existing CA in the ca setting. For example, if your CA is is inlined, it may look something like this:

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    ... existing PEM-encoded CA certificate ...
    -----END NEBULA CERTIFICATE-----
    -----BEGIN NEBULA CERTIFICATE-----
    ... new PEM-encoded CA certificate ...
    -----END NEBULA CERTIFICATE-----
  cert: /etc/nebula/device.crt
  key: /etc/nebula/device.key

If your CA trust bundle exists in a file (e.g. /etc/nebula/ca.crt), the same instructions apply: simply append the new PEM certificate below the existing certificate(s).

In order for this change to take effect, you must restart or reload Nebula. If you're using systemd, systemctl restart nebula will restart the process. Alternatively, to gracefully reload Nebula without tearing down tunnels, find its pid in your process list and run kill -SIGHUP <pid>. You can verify the config was reloaded by looking for the following lines:

time="2022-07-27T11:32:04-04:00" level=info msg="Caught HUP, reloading config"
...
time="2022-07-27T11:32:04-04:00" level=info msg="Trusted CA certificates refreshed" fingerprints=[d5978d6d54a58e4685551708c5f57fbdd3774be67d470ecb0033cf70bbf5fbb5 b680813a506933b2020f6e9980fc26f7df79c8124302e32aedba23c820e813dc]"

For extra credit, you can verify that your newly minted CA's fingerprint appears in the log line shown above.

Step 3: Signing new host certificates​

Now that all of your hosts have been updated to trust the new CA, it's time to issue new host certificates. You'll need to iterate all existing certificates and issue a new certificate using the same information as before:

$ nebula-cert print -path host1.crt
NebulaCertificate {
	Details {
		Name: host1
		Ips: [
			192.168.100.5/24
		]
		Subnets: [
			10.0.0.0/8
		]
		Groups: [
			"prod"
			"api"
        ]
		Not before: 2023-02-30 16:22:00 -0400 EDT
		Not After: 2023-07-30 16:08:16 -0400 EDT
		Is CA: false
		Issuer: d5978d6d54a58e4685551708c5f57fbdd3774be67d470ecb0033cf70bbf5fbb5
		Public key: 4a915591ff1a6869acb085d0292cbd25ba88624b9729420acb20d03644e0b016
		Curve: CURVE25519
	}
	Fingerprint: 92efefd0575f71c10973dc96d9e2111d62703139383855f5a6a74feea68af05e
	Signature: dc680011a11078fc00cce84d176662f54c96fa071d1bd49d5410a987f5743c3a641e27142ec19d5ed1929d5464bcdffe927a787b3a4f200b008d84821e3c4a0d
}
$ nebula-cert sign -name "host1" -subnets "10.0.0.0/8" -ip "192.168.100.5/24" -groups "prod,api"

This will create a new host1.crt and host1.key. Copy the new certificate and key to your host, update the Nebula config (pki.cert and pki.key), and restart or reload Nebula as you did in the previous step.

Even though this host is using a different CA than the rest of your network you should still be able to communicate with the rest of your hosts because its certificate is signed by a CA in their trust bundle. You can issue a ping to test this now. Don't forget that your firewall must allow pings from this host.

Step 4: Removing the old CA from the trust bundle​

You did it! All of your hosts have updated certificates and are communicating with each other. At this point, you can safely remove the original CA from the trust bundle so that only the new CA exists:

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    ... new PEM-encoded CA certificate ...
    -----END NEBULA CERTIFICATE-----
  cert: /etc/nebula/device.crt
  key: /etc/nebula/device.key

Congratulations on rotating your certificate authority. Don't forget to set a reminder before the next expiration!

info

Sick of manually managing IP addresses, certificates, and certificate authority expirations?

Defined Networking can help! We offer a Managed Nebula solution that removes the toil of managing your network. Try it today!