Extend network access beyond overlay hosts

This guide explains how to configure Nebula to route traffic destined for a specific subnet through a specific overlay network host using Nebula's unsafe_routes feature.

This is especially useful for accessing hosts that cannot be modified to run Nebula, such as printers, physical access control systems, and other proprietary devices on which you cannot install arbitrary software.

Prerequisites

Read the Quick Start guide to learn how to create your first overlay network.

You'll need the following to complete this guide.

1. A working overlay network with a lighthouse and at least two hosts:
   • Linux host that will act as the router between the overlay network and local network
   • Linux, macOS, or Windows host that wants to access the local network via an overlay network connection to the "Linux router" host
2. nebula-cert binary to sign host certificates
3. The ca.key and ca.crt files for the working overlay network
4. Root access to a Linux host on the network that will route traffic using unsafe_routes
5. Root access to a Linux, macOS, or Windows host on a different network than the Linux host that will route traffic.

This guide assumes that you have the directory for the nebula and nebula-cert binaries in your $PATH.

You will also need to confirm that your CA is able to sign host certificates with the metadata required to route traffic using unsafe_routes. If you didn't specify -subnets when creating your CA you're good to go. To confirm, run the following from the directory containing your CA cert.

nebula-cert print -path ca.crt

If Subnets is an empty set or if it contains the CIDR that you want to access, you are good to go.

Here's an example of a CA generated without specifying any subnets.

Tip: Add a -json flag to the nebula-cert print command and pipe the output to jq.

nebula-cert print -json -path ca.crt | jq .details
{
  "groups": [],
  "ips": [],
  "isCa": true,
  "issuer": "",
  "name": "BK Labs",
  "notAfter": "2022-12-02T19:21:05-05:00",
  "notBefore": "2021-12-02T19:21:05-05:00",
  "publicKey": "50dd6fb1d2c02f17ddfeb017fe1bf16cf69d42ec28e8a2e02fde5ad2f944f136",
  "subnets": []
}

Example Network

The following IP addresses, hostnames, and subnets are used throughout this guide to illustate a valid configuration for our use case.

Home network (LAN)

This is the subnet that we want to be able to access remotely over our Nebula overlay.

• 192.168.86.0/24 (192.168.86.1–192.168.86.254)
• The Linux host routing traffic from Nebula using unsafe_routes is connected to this network

LAN Host IP	LAN Hostname	Overlay Hostname	Description
192.168.86.10	raspi.lan	home-raspi	Linux host on LAN that will handle routing
192.168.86.5	printer.lan	(none)	Printer that cannot run Nebula

Overlay network (Nebula)

This is the overlay network that will be used by hosts running Nebula.

• 192.168.100.0/24 (192.168.100.1–192.168.100.254)
• The macOS host in this example has Internet access but it not on the same, physical LAN as the Linux host.

Overlay Host IP	Overlay Hostname	Description
192.168.100.10	home-raspi	Linux host that will route traffic between LAN and overlay
192.168.100.11	laptop-mac	Mac that will access printer via home-raspi

Configuration Steps

Using the example network and hosts referenced above, the following steps explain how to configure the macOS host (laptop-mac, 192.168.100.11) to route traffic through the Linux host (home-raspi, 192.168.100.10) in order to reach home printer from anywhere.

Step 1. Sign cert with subnets you want to route

From the same directory containing our CA cert (ca.crt) and key (ca.key), run the following command to sign a new cert for our Linux host that will be routing traffic to the home network.

nebula-cert sign -name 'home-raspi' -ip '192.168.100.10/24' -subnets '192.168.86.0/24'

Inspect the new certificate to confirm that subnets is now correctly set to the Home LAN CIDR.

nebula-cert print -json -path home-raspi.crt | jq .details
{
  "groups": [],
  "ips": [
    "192.168.100.10/24"
  ],
  "isCa": false,
  "issuer": "57903a07e52a8f5464636aeccb1942560324dcd1f6c4f0457d77b00372b5d9f2",
  "name": "home-raspi",
  "notAfter": "2022-12-02T19:21:04-05:00",
  "notBefore": "2021-12-03T11:19:38-05:00",
  "publicKey": "defd2478b4818659d21d862a7dc51a6630fce8fc1fafdec1026c6552f01c0655",
  "subnets": [
    "192.168.86.0/24"
  ]
}

Step 2. Copy cert to Linux host (LAN)

Next, copy the new host certificate and key over to the Linux host that will handle the routing.

scp home-raspi.crt home-raspi.key root@raspi.lan:/etc/nebula

After copying these files, login to this host and replace your existing key and cert with these files. This is commonly /etc/nebula. It's also a best practice to make sure that the key file is only readable by root. If you keep the same file paths as the old key and cert you won't need to make any changes to your Nebula config.yml file.

Step 3. Configure Nebula firewall on Linux host (LAN)

In addition to adding a CIDR to the subnets field of our new cert, you also need to add inbound firewall rules to the Linux host in this example for any port that you want to access through this host.

For example, if you want to access a printer on the LAN, you will need to add a rule to allow TCP port 631. Edit the Linux host's Nebula configuration file (commonly located at /etc/nebula/config.yml).

firewall:
  inbound:
    - port: 631
      host: any
      proto: tcp

You may need to add additional ports depending on your printer. The easiest way to make sure you can fully access any host on your network is to add a rule to allow any port/protocol on this host. The entire firewall section should look like the following.

firewall:
  inbound:
    - port: any
      host: any
      proto: any
  outbound:
    - host: any
      port: any
      proto: any

At this point we're finished configuring the Nebula-specific components on our Linux host. If Nebula is already running, go ahead and stop it. It's a good idea to start it again to validate that the configuration file, cert, and key are set up properly.

sudo nebula -config /etc/nebula/config.yml
INFO[0000] Firewall rule added              firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:any ip: proto:0 startPort:0]"
INFO[0000] Firewall rule added              firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: proto:0 startPort:0]"
INFO[0000] Firewall started                 firewallHash=21716b47a7a140e448077fe66c31b4b42f232e996818d7dd1c6c4991e066dbdb
INFO[0000] Main HostMap created             network=192.168.100.10/24 preferredRanges="[]"
INFO[0000] UDP hole punching enabled
INFO[0000] Nebula interface is active       build=1.5.0 interface=nebula network=192.168.100.10/24 udpAddr="[::]:43068"

If you see a Handshake message sent to your lighthouse followed by a corresponding received message after a similar set of lines shown above you are good to go.

You can either leave Nebula running in the background or stop it while completing the next steps specific to this host.

Step 4. Enable IP forwarding on Linux host (LAN)

Linux hosts need a kernel parameter set in order to allow packet forwarding. This is not typically enabled by default as shown in the following read example.

sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

Here's how you update that variable at runtime.

sudo sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

Note: This change is only persistent until you reboot. To make it permanent, add a new line with net.ipv4.ip_forward = 1 to the end of the /etc/sysctl.conf file.

Step 5. Configure packet routing on Linux host (LAN)

+Now that IP forwarding is enabled, we need to add a few firewall rules so that our Linux router host will be able to to act as a NAT and masquerade as the other Nebula nodes which are using the route to connect through the Linux host to the hosts in the VPC. The exact steps necessary will depend on the firewall you are using, and the current firewall state on the host.

ufw

Open /etc/ufw/before.rules and add the following lines to the bottom of the file.

# NAT table rules
*nat
:POSTROUTING ACCEPT [0:0]

# flush all existing POSTROUTING rules in the nat table to avoid duplicates
-F POSTROUTING

# 192.168.100.0/24 is the Nebula network, 192.168.86.0/24 is the route CIDR
-A POSTROUTING -s 192.168.100.0/24 -d 192.168.86.0/24 -j MASQUERADE

# don't delete the 'COMMIT' line or these nat table rules won't be processed
COMMIT

Then run the following commands, replacing nebula1 (Nebula interface) and enp5s0 (local network interface) with the correct interface names, if necessary:

sudo ufw route allow in on nebula1 out on enp5s0 from 192.168.100.0/24 to 192.168.86.0/24
sudo ufw reload

Confirm the new rule is active by running the following:

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
192.168.86.0/24 on enp5s0  ALLOW FWD   192.168.100.0/24 on nebula1

nftables

Create a new nftables configuration (e.g. /etc/nftables-routing.conf) with the following configuration:

table ip nebula_routing
delete table ip nebula_routing
table ip nebula_routing {
  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    ip saddr 192.168.100.0/24 ip daddr 192.168.86.0/24 counter masquerade
  }

  chain forward {
    type filter hook forward priority filter; policy accept;
    ct state related,established counter accept
    iifname nebula1 oifname enp6s0 ip saddr 192.168.100.0/24 ip daddr 192.168.86.0/24 counter accept
  }
}

Then run the following command to load the rules:

sudo nft -f /etc/nftables-routing.conf

You can verify the rule has taken effect by running the following command:

$ sudo nft list table nebula_routing
table ip nebula_routing {
	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 192.168.100.0/24 ip daddr 192.168.86.0/24 counter packets 1 bytes 84 masquerade
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		ct state established,related counter packets 134763 bytes 123879509 accept
		iifname "nebula1" oifname "enp6s0" ip saddr 192.168.100.0/24 ip daddr 192.168.86.0/24 counter packets 726 bytes 85482 accept
	}
}

note

These rules will only be persistent until you reboot the host. You will need to configure the file to be loaded at boot.

iptables

Run the following commands to add the rules specific to our example networks, replacing nebula1 (Nebula interface) and epn5s0 (local network interface) with the correct interface names, if necessary.

sudo iptables -t nat -A POSTROUTING -s 192.168.100.0/24 -d 192.168.86.0/24 -j MASQUERADE
sudo iptables -I FORWARD 1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -I FORWARD 2 -i nebula1 -o enp5s0 -s 192.168.100.0/24 -d 192.168.86.0/24 -j ACCEPT

Once complete, you can confirm that the new rules are active by running the following.

This first list shows the second and third rule.

$ sudo iptables -vL
Chain INPUT (policy ACCEPT 10M packets, 6461M bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 86 packets, 11057 bytes)
 pkts bytes target     prot opt in     out     source               destination
 1034 1469K ACCEPT     all  --  epn6s0 nebula1  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  nebula1 enp6s0  192.168.100.0/24     192.168.86.0/24

Chain OUTPUT (policy ACCEPT 18M packets, 58G bytes)
 pkts bytes target     prot opt in     out     source               destination

And this command shows the first rule, specific to the NAT table.

$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  192.168.100.0/24     192.168.86.0/24

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

You may see additional rules listed depending on your host and whether or not you've modified it.

note

These rules will only be persistent until you reboot the host. Depending on your distribution, you need to run additional commands (e.g. iptables-save) to save these rules to disk and load them at boot.

Step 6. Edit Nebula config on the overlay hosts that need to access the home LAN

We're almost there!

The final step in this process is to configure the overlay network hosts to use unsafe_routes and route traffic destined for our home LAN through the Linux host that we just configured.

Keeping with the example Overlay network in this guide, we will add the unsafe_routes details to the tun section of the Nebula config.yml file on laptop-mac.

tun:
  dev: utun10
  drop_local_broadcast: true
  drop_multicast: true
  mtu: 1300
  unsafe_routes:
    - route: 192.168.86.0/24
      via: 192.168.100.10

Run Nebula with the new configuration:

sudo nebula -config config.yml
INFO[0000] Firewall rule added           firewallRule="map[caName: caSha: direction:outgoing endPort:0 groups:[] host:any ip: proto:0 startPort:0]"
INFO[0000] Firewall rule added           firewallRule="map[caName: caSha: direction:incoming endPort:0 groups:[] host:any ip: proto:1 startPort:0]"
INFO[0000] Firewall started              firewallHash=570dd0546e17b139da845c05717d6dc2005fa7083292e1e8f797affab434c4f4
WARN[0000] Adding UNSAFE Route           route=192.168.86.0/24 via=192.168.100.10
INFO[0000] Main HostMap created          network=192.168.100.11/24 preferredRanges="[]"
INFO[0000] UDP hole punching enabled
INFO[0000] Nebula interface is active    build=1.5.0 interface=utun10 network=192.168.100.11/24 udpAddr="[::]:57027"

You should see a WARN line above indicating that we have an UNSAFE route being used. And now... you should be able to successfully ping any host on the 192.168.86.0/24 Home LAN.

Ping working? Great! Go ahead and add that printer, map file shares, and VNC to any host on that LAN.

Add the same unsafe_routes section to any other Nebula host that needs to access hosts on the Home LAN that aren't running Nebula.

Tips

Avoid IP conflicts with underlay networks

When you are running Nebula with unsafe_routes the OS of the host will forward all traffic destined for the route specified to Nebula. Keep in mind that this could cause a local IP conflict if your underlay network happens to use the same subnet as the subnet specified in your unsafe_routes configuration.

Works with Mobile Nebula

After you've successfully completed the steps above, you can also access your extended networks from a device running Mobile Nebula on Android or iOS by launching the app and configuring it as follows:

1. Tap the site you'd like to configure
2. Then tap Configuration → Advanced → Unsafe routes
3. Tap Add a new route and enter the same information you used in Step 6:
   • Route: 192.168.86.0/24
   • Via: 192.168.100.10
4. Tap Save

If already connected to the site, disconnect.

Connect and try to access one of the hosts only accessible using unsafe_routes. You should see a new line in your Mobile Nebula logs.

level=warning msg="Adding UNSAFE Route" route=192.168.86.0/24 via=192.168.100.10