How to Build a Network Automation Stack with Nornir, Napalm and Netbox

Introduction

In this article, we will provide a hands-on guide on how to build a network automation stack using the open-source tools - Nornir, Napalm, and Netbox. In addition, we will then leverage these tools to dive into 2 examples:

Backup Configurations - Logging into each device, pulling the running configuration and saving it locally, using Netbox as the inventory.
Populating Netbox - Retrieving data from the network and then populating Netbox using this information.

Note: The accompanying code for this article is located at https://github.com/rickdonato/nornir-napalm-netbox-demo

Components

Our guide will be based upon the integration of 3 open source tools - Nornir, Napalm, and Netbox. Let’s look at each of the tools.

Netbox

Netbox is an open-source application, built upon the Python Django framework, designed to help manage and document computer networks.^[1] Netbox provides a good UI allowing you to visualize your racks, a REST API, and many other great features such as webhooks and export templating to name but a few. In addition, Netbox encompasses the following aspects of network management:

IP address management (IPAM) - IP networks and addresses, VRFs, and VLANs
Equipment racks - Organized by group and site
Devices - Types of devices and where they are installed
Connections - Network, console, and power connections among devices
Virtualization - Virtual machines and clusters
Data circuits - Long-haul communications circuits and providers
Secrets - Encrypted storage of sensitive credentials

Figure 1 - Netbox UI.

Napalm

NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) is a Python library that implements a set of functions to interact with different network device Operating Systems using a unified API.

Figure 2 - Napalm

In other words, NAPALM abstracts the lower level semantics of device interaction, such as which backend device library should be used (pyeapi, pynxos, netmiko etc), and which calls to make. Napalm then presents a common set of methods that are agnostic of the platform type.

Let’s look at an example. Let's say we want to collect the serial number of a Junos and an IOS based device. This can be performed by initializing a Napalm driver for the network os type and performing the get_facts() method. As you can see below the input method (get_facts) and also the output response structure is the same for both devices, regardless of OS type.

>>> driver = get_network_driver("junos")
>>> device = driver(hostname="172.29.133.2", username="admin", password="Juniper")
>>> device.open()
>>> pprint(device.get_facts())
{'fqdn': 'vqfx',
 'hostname': 'vqfx',
 'interface_list': ['gr-0/0/0',
                    'pfe-0/0/0',
                    'pfh-0/0/0',
                    'xe-0/0/0',
                    'xe-0/0/1',
                    'xe-0/0/2',
                    'xe-0/0/3',
                    'xe-0/0/4',
                    '.local.',
                    'bme0',
                    'cbp0',
                    'dsc',
                     ...],
 'model': 'VQFX-10000',
 'os_version': '18.1R3-S2.5',
 'serial_number': '25723514507',
 'uptime': 1684,
 'vendor': 'Juniper'}

>>> driver = get_network_driver("ios")
>>> device = driver(hostname="172.29.133.3", username="cisco", password="cisco")
>>> device.open()
>>> pprint(device.get_facts())
{'fqdn': 'vios.packetflow.local',
 'hostname': 'vios',
 'interface_list': ['GigabitEthernet0/0',
                    'GigabitEthernet0/1',
                    'GigabitEthernet0/2',
                    'GigabitEthernet0/3',
                    'GigabitEthernet1/0',
                    'GigabitEthernet1/1',
                    'GigabitEthernet1/2',
                    'GigabitEthernet1/3'],
 'model': 'IOSv',
 'os_version': 'vios_l2 Software (vios_l2-ADVENTERPRISEK9-M), Experimental '
               'Version 15.2(20170321:233949) [mmen 101]',
 'serial_number': '9E5DPLAM2DE',
 'uptime': 1680,
 'vendor': 'Cisco'}

Nornir

Nornir (from the creator of Napalm, David Barrasso) is a pluggable multi-threaded framework with inventory management to help operate collections of devices.^[2] Nornir, is 100% Python. Whereas many automation frameworks (such as Ansible) use their own Domain Specific Language (DSL), Nornir lets you control everything from Python. Why does being 100% Python matter?

You can use your existing IDE for code development.
Easier to troubleshoot and debug by using existing Python logging and debugging tools.
There is no DSL, which can get complex when advanced features are added.

We have defined, at a high level, what Nornir is, but let's dive into the actual details and the various components that make up the Nornir automation framework.

Tasks - A Task is a Python function that is run on a per-host basis.
Functions - Functions run globally. For example print_result is a function that prints the results of executed tasks.
Inventory - Defines the devices (and any relating attributes) that our tasks will be performed against. The inventory structure consists of:
Hosts - hosts.yaml - Individual hosts and any related attributes.
Groups - groups.yaml - Group definitions and any related attributes for the groups. Hosts are then added to these groups within hosts.yaml.
Defaults - defaults.yaml - Contains attributes that can be assigned to all hosts regardless of group assignment.

Credit: Appreciation goes out to the creators of these open source tools, their great work, and continued work in the network automation community.

David Barrasso - Nornir/Napalm.
Jeremy Stretch - Netbox.
Kirk Byers - Netmiko.

Stack Overview

Now that we have an understanding of each of the components, let's turn our attention to how the components will integrate (at a high level) which each other.

Netbox will run as a Docker container.
Napalm will be consumed within Nornir via the nornir.plugins.connections.napalm.Napalm plugin.
Nornir and its plugins will run within a python virtual environment.
Nornir will pull the inventory data from Netbox via the REST API.

Figure 3 - High Level overview of Nornir, Napalm and Netbox Integration.

In terms of our topology for this demo we will be using a topology consisting of the following devices:

Juniper QFX
Cisco Switch
Arista Switch

Installation

Netbox

To install Netbox run the following commands on your Docker host.

git clone -b master https://github.com/netbox-community/netbox-docker.git
cd netbox-docker
docker-compose pull
docker-compose up -d

Once installed you should see your running Netbox containers, like so:

~# docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
f1f2a0d0aed6        nginx:1.15-alpine               "nginx -c /etc/netbo…"   8 days ago          Up 8 days           80/tcp, 0.0.0.0:32768->8080/tcp                            netboxdocker_nginx_1
3b246d15de04        netboxcommunity/netbox:latest   "/opt/netbox/docker-…"   8 days ago          Up 8 days                                                                      netboxdocker_netbox_1
a916283fd64b        netboxcommunity/netbox:latest   "python3 /opt/netbox…"   8 days ago          Up 8 days                                                                      netboxdocker_netbox-worker_1
36cf310e5def        postgres:10.4-alpine            "docker-entrypoint.s…"   8 days ago          Up 8 days           5432/tcp                                                   netboxdocker_postgres_1
c013f8e5c424        redis:4-alpine                  "docker-entrypoint.s…"   8 days ago          Up 8 days           6379/tcp                                                   netboxdocker_redis_1

For full details around running Netbox on Docker go to: https://github.com/netbox-community/netbox-docker.

Nornir/Napalm

To install Nornir and Napalm we need to create a Python virtual environment and then perform pip install … for each of the packages.

For those of you new to virtual environments,

Virtual environments helps to keep dependencies required by different projects separated by creating isolated Python virtual environments for them.^[3]

However for this article, I've added the required commands to a Makefile to make things slightly easier. Therefore, the only commands to get you up and running are shown below,

git clone https://github.com/rickdonato/nornir-napalm-netbox-demo
cd nornir-napalm-netbox-demo/
make add-venv-py3.6

Once complete, your ls -l will look like the below.

.
|-- Makefile
|-- README.md
|-- config.yaml
|-- data
|   `-- configs
|-- requirements.txt
|-- scripts
|   |-- __init__.py
|   |-- backup_configs.py
|   |-- helpers.py
|   |-- secrets.py
|   |-- create_interfaces.py
|   `-- update_interfaces.py
`-- venv

Note: For those of you who haven't used Makefiles before they simply allow us to bundle a number of shell commands together and run them via a single make statement. If you want to learn more, feel free to either perform a cat Makefile to see the contents or for a more detailed description of Makefile click here.

To validate Nornir and Napalm are installed check the installed packages via pip freeze, like so:

root@desktop:/tmp/nornir-napalm-netbox-demo# source venv/bin/activate
(venv) root@desktop:/tmp/nornir-napalm-netbox-demo# pip freeze | grep -E "nornir|napalm"
napalm==2.4.0
nornir==2.3.0

Configuration

Creating Devices in Netbox

You will need to create your devices within Netbox. The step-by-step instructions on how to do this is outside the scope of this document but at a high level, you will need to:

Create Manufacturers under Device Types.
Create Device Types under Device Types.
Create Platforms under Devices.
Create Device Roles under Devices.
Create Devices under Devices.
Within each device:
1. add a management interface.
2. mark management interface as primary.
3. assign an IP.

Figure 4 - Adding Devices to Netbox.

Nornir-to-Netbox Configuration

We will now turn our attention to integrating Netbox with Nornir via the Netbox plugin.

First of all create an API Token via the Netbox UI like so,

Figure 5 - Create Netbox API Token.

Next, edit config.yaml (shown below) adding your Netbox API Token and Netbox host details.

---
core:
    num_workers: 100

inventory:
  plugin: nornir.plugins.inventory.netbox.NBInventory
  options:
    nb_url: 'http://<NETBOX_HOST>:32768'
    nb_token: '<NETBOX_API_TOKEN>'
    ssl_verify: False
  transform_function: "helpers.adapt_user_password"

Unfortunately, the Nornir Netbox plugin doesn't support Netbox secrets, therefore to inject the required credentials into our inventory we will use a transform function. Our transform function (shown below) will perform a key lookup from scripts/secrets.py for the username and password and then reassign the host.username and host.password inventory attributes.

scripts/helpers.py

from secrets import creds

def adapt_user_password(host):
    host.username = creds[f"{host}"]["username"]
    host.password = creds[f"{host}"]["password"]

scripts/secrets.py

creds = {
    "vqfx": {"username": "admin", "password": "Juniper"},
    "vios": {"username": "cisco", "password": "cisco"},
    "veos": {"username": "admin", "password": "arista"},
}

Note: Although it goes without saying, having your passwords in a plain text file (for production) is not recommended. Therefore, in a production based environment I would recommend to use a password management tool like Vault. The integration required for this, however, is outside of the scope of this article.

With all the previous configuration complete we can now test that everything is working by printing the inventory, via scripts/helpers.py --inventory:

# scripts/helpers.py --inventory
{'defaults': {'connection_options': {},
              'data': {},
              'hostname': None,
              'password': None,
              'platform': None,
              'port': None,
              'username': None},
 'groups': {},
 'hosts': {'veos': {'connection_options': {},
                    'data': {'asset_tag': None,
                             'model': 'eos',
                             'role': 'switch',
                             'serial': '',
                             'site': 'site1',
                             'vendor': 'Arista'},
                    'groups': [],
                    'hostname': '172.29.133.4',
                    'password': 'arista',
                    'platform': 'eos',
                    'port': None,
                    'username': 'admin'},
           'vios': {'connection_options': {},
                    'data': { 'asset_tag': None,
                             'model': 'ios',
                             'role': 'switch',
                             'serial': '',
                             'site': 'site1',
                             'vendor': 'Cisco'},
                    'groups': [],
                    'hostname': '172.29.133.3',
                    'password': 'cisco',
                    'platform': 'ios',
                    'port': None,
                    'username': 'cisco'},
...

Examples

With the previous configuration complete, let's turn our attention to our examples.

Backup Configurations

Overview

First we will build a script (scripts/backup_configs.py) that will leverage a custom Nornir task to perform a backup of our network device configurations, saving the configs to a local directory.

In the real world this example/script could be extended to run as part on a regular basis (via crontab) or imported as a module. In addition you could also add additional logic to push this up to a code repo such as Github.

In order to achieve this will build a single Python file. Like so:

#!./venv/bin/python

from nornir.plugins.tasks import networking
from nornir.plugins.functions.text import print_result
from nornir.plugins.tasks.files import write_file
from nornir import InitNornir

BACKUP_PATH = "./data/configs"


def backup_config(task, path):
    r = task.run(task=networking.napalm_get, getters=["config"])
    task.run(
        task=write_file,
        content=r.result["config"]["running"],
        filename=f"{path}/{task.host}.txt",
    )


nr = InitNornir(config_file="./config.yaml")

devices = nr.filter(role="switch")

result = devices.run(
    name="Backup Device Configurations", path=BACKUP_PATH, task=backup_config
)

print_result(result, vars=["stdout"])

Code Explanation

Let us go through the script on some more detail.

First we perform our imports and set the location where we want to save our configurations to.

from nornir.plugins.tasks import networking
from nornir.plugins.functions.text import print_result
from nornir.plugins.tasks.files import write_file
from nornir import InitNornir
 
BACKUP_PATH = "./data/configs"

Next we create a task. This task performs 2 main actions:

Napalm Getter - Using the napalm plugin it runs the config (aka get_config) Napalm getter.
Write File - The write_file plugin saves the result into our intended destination.

def backup_config(task, path):
    r = task.run(task=networking.napalm_get, getters=["config"])
    task.run(
        task=write_file,
        content=r.result["config"]["running"],
        filename=f"{path}/{task.host}.txt",
    )

We initialise Nornir against the configuration file and then filter the devices we are interested in (i.e we will later run the task against). Remember the role selected in the filter is the role defined/assigned within Netbox.

nr = InitNornir(config_file="./config.yaml")

devices = nr.filter(role="switch")

Finally we bring it all together. We run our task backup_config against our devices object, also passing in any variables that the task requires. In our case BACKUP_PATH.

Once done, we print the results out to screen.

result = devices.run(
    name="Backup Device Configurations", path=BACKUP_PATH, task=backup_config
)

print_result(result, vars=["stdout"])

Example

Updating Netbox

Overview

Our next example will be based on populating Netbox using data collected from our devices.

This will be a little bit more involved, but in essence is based on 2 steps:

Create Interface - scripts/create_interfaces.py - Pulling the interface data from the device and creating the related interfaces.
Update Interface - scripts/update_interfaces.py - Pulling the interface data from the device and updating the corresponding interfaces in Netbox.

To update Netbox we will use python-netbox (read more). Let’s look at the code:

scripts/create_interfaces.py

import re
from nornir.plugins.tasks import networking
from nornir.plugins.functions.text import print_result
from nornir import InitNornir
from netbox import NetBox
from helpers import get_device_id, is_interface_present
import pprint

nr = InitNornir(config_file="./config.yaml")

nb_url, nb_token, ssl_verify = nr.config.inventory.options.values()
nb_host = re.sub("^.*//|:.*$", "", nb_url)

netbox = NetBox(host=nb_host, port=32768, use_ssl=False, auth_token=nb_token)

nb_interfaces = netbox.dcim.get_interfaces()

def create_netbox_interface(task, nb_interfaces, netbox):
    r = task.run(task=networking.napalm_get, getters=["interfaces"])
    interfaces = r.result["interfaces"]

    for interface_name in interfaces.keys():
        if not is_interface_present(nb_interfaces, f"{task.host}", interface_name):
            print(
                f"* Creating Netbox Interface for device {task.host}, interface {interface_name}"
            )
            device_id = get_device_id(f"{task.host}", netbox)
            netbox.dcim.create_interface(
                name=f"{interface_name}",
                form_factor=1200,  # default
                device_id=device_id,
            )

devices = nr.filter(role="switch")

result = devices.run(
    name="Create Netbox Interfaces",
    nb_interfaces=nb_interfaces,
    netbox=netbox,
    task=create_netbox_interface,
)
print_result(result, vars=["stdout"])

scripts/update_interfaces.py

import re
from nornir.plugins.tasks import networking
from nornir.plugins.functions.text import print_result
from nornir import InitNornir
from netbox import NetBox
from helpers import is_interface_present

nr = InitNornir(config_file="./config.yaml")

nb_url, nb_token, ssl_verify = nr.config.inventory.options.values()
nb_host = re.sub("^.*//|:.*$", "", nb_url)

netbox = NetBox(host=nb_host, port=32768, use_ssl=False, auth_token=nb_token)

nb_interfaces = netbox.dcim.get_interfaces()


def update_netbox_interface(task, nb_interfaces):
    r = task.run(task=networking.napalm_get, getters=["interfaces"])
    interfaces = r.result["interfaces"]

    for interface_name in interfaces.keys():
        mac_address = interfaces[interface_name]["mac_address"]
        if mac_address == "None" or mac_address == "Unspecified":
            mac_address = "ee:ee:ee:ee:ee:ee"

        description = interfaces[interface_name]["description"]

        if is_interface_present(nb_interfaces, f"{task.host}", interface_name):
            print(
                f"* Updating Netbox Interface for device {task.host}, interface {interface_name}"
            )
            netbox.dcim.update_interface(
                device=f"{task.host}",
                interface=interface_name,
                description=description,
                mac_address=mac_address,
            )

devices = nr.filter(role="switch")

result = devices.run(
    name="Update Netbox Interfaces",
    nb_interfaces=nb_interfaces,
    task=update_netbox_interface,
)
print_result(result, vars=["stdout"])

Code Explanation

Within both scripts once we perform the initial imports, we pull the Netbox config details from the Nornir inventory which we then pass in at the point we instantiate netbox.

nb_url, nb_token, ssl_verify = nr.config.inventory.options.values() \
nb_host = re.sub("^.*//|:.*$", "", nb_url) \
 \
netbox = NetBox(host=nb_host, port=32768, use_ssl=False, auth_token=nb_token)

The overall layout is the same as the previous backup configuration example. i.e a task is created, we filter the devices we require and then run our task against the devices, passing in our variables.

In terms of the task logic they are somewhat similar:

scripts/create_interfaces.py

We pull the interfaces from Netbox via netbox.dcim.get_interfaces().
We iterate over the interface dict() keys that are returned.
For each interface we check to see if the interface is already in Netbox via our helper is_interface_present.
If the interface is not present we perform a netbox.dcim.create_interface().

scripts/update_interfaces.py

We pull the interfaces from Netbox via netbox.dcim.get_interfaces().
We iterate over the interface dict() keys that are returned.
For each interface we check to see if the interface is already in Netbox via our helper is_interface_present.
If the interface is present we perform a netbox.dcim.update_interface().
The data we populate is the description and MAC address.

Example

Figure 6 - Populated Netbox data.

Outro

That concludes this series around Nornir, Napalm and Netbox. I can honestly say this article has been a blast to create. As you can see these tools, give you enormous potential and flexibility into how you automate your network. Abstracting many of the lower level details that would typically take a lot of time and patience to develop.

References

"NetBox - Read the Docs." https://netbox.readthedocs.io/. Accessed 23 Oct. 2019. ↩︎
"nornir-automation/nornir: Pluggable multi-threaded ... - GitHub." https://github.com/nornir-automation/nornir. Accessed 23 Oct. 2019. ↩︎
"Python Virtual Environment | Introduction - GeeksforGeeks." https://www.geeksforgeeks.org/python-virtual-environment/. Accessed 26 Oct. 2019. ↩︎

How to Build a Network Automation Stack with Nornir, Napalm and Netbox

Introduction

Components

Netbox

Napalm

Nornir

Stack Overview

Installation

Netbox

Nornir/Napalm

Configuration

Creating Devices in Netbox

Nornir-to-Netbox Configuration

Examples

Backup Configurations

Overview

Code Explanation

Example

Updating Netbox

Overview

Code Explanation

Example

Outro

References

Comments

A Beginners Guide to Docker and VS Code (Part 2)

A Beginners Guide to Docker and VS Code (Part 1)

Network Automation 101 - Skills and Technologies

Network Automation 101 - Hands-on with Python, Jinja and YAML

Network Automation 101 - Tooling Landscape

What is Idempotency?

Introduction

Components

Netbox

Napalm

Nornir

Stack Overview

Installation

Netbox

Nornir/Napalm

Configuration

Creating Devices in Netbox

Nornir-to-Netbox Configuration

Examples

Backup Configurations

Overview

Code Explanation

Example

Updating Netbox

Overview

Code Explanation

Example

Outro

References

Comments

Search PacketFlow