Debugging with Nebula SSH commands

This guide describes useful commands built into the SSH server accessible over nebula, which can allow debugging network connectivity for the nebula host.

First generate a new SSH key for the host you want to debug, via ssh-keygen -t ed25519 -f ssh_host_ed25519_key. You can set it to only accessible by root via chown root:root ssh_host_ed25519_key, which will ensure that regular users on that host cannot access the private key.

Next configure the sshd section on the host you want to debug.

Example config:

sshd:
  enabled: true
  listen: 127.0.0.1:2222
  host_key: /path/to/ssh_host_ed25519_key
  authorized_users:
    - user: steeeeve
      keys:
        - '[ssh public key string]'

In this case steeeeve and [ssh public key string] should be the values for the user you want to enable access for debugging. You must add the correct public ssh key for the users you wish to access the ssh server with. If you don't already have an SSH key for the host you want to access from, follow this guide by GitHub: Generating a new SSH key and adding it to the ssh-agent.

Set the listen property to localhost for local debugging or to your nebula IP to enable access over the nebula overlay network.

Next, either run kill -HUP <nebula PID> or restart nebula and ssh via ssh 127.0.0.1 -p 2222 for example.

Now you can debug your nebula installation, querying things like your hostmap:

> list-hostmap
10.128.1.1: [205.133.20.81:4242]
10.128.1.2: [153.231.31.111:4242]

You should be able to run help once you're logged in.

steeeeve@nebula > help
Available commands:
change-remote - Changes the remote address used in the tunnel for the provided vpn ip
close-tunnel - Closes a tunnel for the provided vpn ip
create-tunnel - Creates a tunnel for the provided vpn ip and address
help - prints available commands or help <command> for specific usage info
list-hostmap - List all known previously connected hosts
list-lighthouse-addrmap - List all lighthouse map entries
list-pending-hostmap - List all handshaking hosts
log-format - Gets or sets the current log format
log-level - Gets or sets the current log level
logout - Ends the current session
mutex-profile-fraction - Gets or sets runtime.SetMutexProfileFraction
print-cert - Prints the current certificate being used or the certificate for the provided vpn ip
print-relays - Prints json details about all relay info
print-tunnel - Prints json details about a tunnel for the provided vpn ip
query-lighthouse - Query the lighthouses for the provided vpn ip
reload - Reloads configuration from disk, same as sending HUP to the process
save-heap-profile - Saves a heap profile to the provided path
save-mutex-profile - Saves a mutex profile to the provided path
start-cpu-profile - Starts a cpu profile and write output to the provided file
stop-cpu-profile - Stops a cpu profile and writes output to the previously provided file
version - Prints the currently running version of nebula

You can discover additional information about each command by running help <command>.

> help list-hostmap
list-hostmap - List all known previously connected hosts
  -by-index
    	gets all hosts in the hostmap from the index table
  -json
    	outputs as json with more information
  -pretty
    	pretty prints json, assumes -json

Notes about some commands

query-lighthouse <some-ip> will return an empty result set initially if the host is not connected, but it will trigger a background request to the Lighthouse. Meaning, you need to run it twice to actually get a result.

change-remote has only a temporary effect: after a period of time, Nebula will "revert" to its preferred remote

Debugging with Nebula SSH commands

Notes about some commands​

Notes about some commands