Introduction
In this article, we will provide a hands-on guide on how to build a network automation stack using the open-source tools - Nornir, Napalm, and Netbox. In addition, we will then leverage these tools to dive into 2 examples:
- Backup Configurations - Logging into each device, pulling the running configuration and saving it locally, using Netbox as the inventory.
- Populating Netbox - Retrieving data from the network and then populating Netbox using this information.
Note: The accompanying code for this article is located at https://github.com/rickdonato/nornir-napalm-netbox-demo
Components
Our guide will be based upon the integration of 3 open source tools - Nornir, Napalm, and Netbox. Let’s look at each of the tools.
Netbox
Netbox is an open-source application, built upon the Python Django framework, designed to help manage and document computer networks.[1] Netbox provides a good UI allowing you to visualize your racks, a REST API, and many other great features such as webhooks and export templating to name but a few. In addition, Netbox:
encompasses the following aspects of network management:
- IP address management (IPAM) - IP networks and addresses, VRFs, and VLANs
- Equipment racks - Organized by group and site
- Devices - Types of devices and where they are installed
- Connections - Network, console, and power connections among devices
- Virtualization - Virtual machines and clusters
- Data circuits - Long-haul communications circuits and providers
- Secrets - Encrypted storage of sensitive credentials
Figure 1 - Netbox UI.
Napalm
NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support) is a Python library that implements a set of functions to interact with different network device Operating Systems using a unified API.
Figure 2 - Napalm
In other words, NAPALM abstracts the lower level semantics of device interaction, such as which backend device library should be used (pyeapi, pynxos, netmiko etc), and which calls to make. Napalm then presents a common set of methods that are agnostic of the platform type.
Let’s look at an example. Let's say we want to collect the serial number of a Junos and an IOS based device. This can be performed by initializing a Napalm driver for the network os type and performing the get_facts() method. As you can see below the input method (get_facts) and also the output response structure is the same for both devices, regardless of OS type.
>>> driver = get_network_driver("junos")
>>> device = driver(hostname="172.29.133.2", username="admin", password="Juniper")
>>> device.open()
>>> pprint(device.get_facts())
{'fqdn': 'vqfx',
'hostname': 'vqfx',
'interface_list': ['gr-0/0/0',
'pfe-0/0/0',
'pfh-0/0/0',
'xe-0/0/0',
'xe-0/0/1',
'xe-0/0/2',
'xe-0/0/3',
'xe-0/0/4',
'.local.',
'bme0',
'cbp0',
'dsc',
...],
'model': 'VQFX-10000',
'os_version': '18.1R3-S2.5',
'serial_number': '25723514507',
'uptime': 1684,
'vendor': 'Juniper'}
>>> driver = get_network_driver("ios")
>>> device = driver(hostname="172.29.133.3", username="cisco", password="cisco")
>>> device.open()
>>> pprint(device.get_facts())
{'fqdn': 'vios.packetflow.local',
'hostname': 'vios',
'interface_list': ['GigabitEthernet0/0',
'GigabitEthernet0/1',
'GigabitEthernet0/2',
'GigabitEthernet0/3',
'GigabitEthernet1/0',
'GigabitEthernet1/1',
'GigabitEthernet1/2',
'GigabitEthernet1/3'],
'model': 'IOSv',
'os_version': 'vios_l2 Software (vios_l2-ADVENTERPRISEK9-M), Experimental '
'Version 15.2(20170321:233949) [mmen 101]',
'serial_number': '9E5DPLAM2DE',
'uptime': 1680,
'vendor': 'Cisco'}
Nornir
Nornir (from the creator of Napalm, David Barrasso) is a pluggable multi-threaded framework with inventory management to help operate collections of devices.[2] Nornir, is 100% Python. Whereas many automation frameworks (such as Ansible) use their own Domain Specific Language (DSL), Nornir lets you control everything from Python. Why does being 100% Python matter?
- You can use your existing IDE for code development.
- Easier to troubleshoot and debug by using existing Python logging and debugging tools.
- There is no DSL, which can get complex when advanced features are added.
We have defined, at a high level, what Nornir is, but let's dive into the actual details and the various components that make up the Nornir automation framework.
- Tasks - A Task is a Python function that is run on a per-host basis.
- Functions - Functions run globally. For example
print_resultis a function that prints the results of executed tasks. - Inventory - Defines the devices (and any relating attributes) that our tasks will be performed against. The inventory structure consists of:
- Hosts -
hosts.yaml- Individual hosts and any related attributes. - Groups -
groups.yaml- Group definitions and any related attributes for the groups. Hosts are then added to these groups withinhosts.yaml. - Defaults -
defaults.yaml- Contains attributes that can be assigned to all hosts regardless of group assignment.
Credit: Appreciation goes out to the creators of these open source tools, their great work, and continued work in the network automation community.
- David Barrasso - Nornir/Napalm.
- Jeremy Stretch - Netbox.
- Kirk Byers - Netmiko.
Stack Overview
Now that we have an understanding of each of the components, let's turn our attention to how the components will integrate (at a high level) which each other.
- Netbox will run as a Docker container.
- Napalm will be consumed within Nornir via the
nornir.plugins.connections.napalm.Napalmplugin. - Nornir and its plugins will run within a python virtual environment.
- Nornir will pull the inventory data from Netbox via the REST API.
Figure 3 - High Level overview of Nornir, Napalm and Netbox Integration.
In terms of our topology for this demo we will be using a topology consisting of the following devices:
- Juniper QFX
- Cisco Switch
- Arista Switch
Installation
Netbox
To install Netbox run the following commands on your Docker host.
git clone -b master https://github.com/netbox-community/netbox-docker.git
cd netbox-docker
docker-compose pull
docker-compose up -d
Once installed you should see your running Netbox containers, like so:
~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f1f2a0d0aed6 nginx:1.15-alpine "nginx -c /etc/netbo…" 8 days ago Up 8 days 80/tcp, 0.0.0.0:32768->8080/tcp netboxdocker_nginx_1
3b246d15de04 netboxcommunity/netbox:latest "/opt/netbox/docker-…" 8 days ago Up 8 days netboxdocker_netbox_1
a916283fd64b netboxcommunity/netbox:latest "python3 /opt/netbox…" 8 days ago Up 8 days netboxdocker_netbox-worker_1
36cf310e5def postgres:10.4-alpine "docker-entrypoint.s…" 8 days ago Up 8 days 5432/tcp netboxdocker_postgres_1
c013f8e5c424 redis:4-alpine "docker-entrypoint.s…" 8 days ago Up 8 days 6379/tcp netboxdocker_redis_1
For full details around running Netbox on Docker go to: https://github.com/netbox-community/netbox-docker.
Nornir/Napalm
To install Nornir and Napalm we need to create a Python virtual environment and then perform pip install … for each of the packages.
For those of you new to virtual environments,
Virtual environments helps to keep dependencies required by different projects separated by creating isolated Python virtual environments for them.[3]
However for this article, I've added the required commands to a Makefile to make things slightly easier. Therefore, the only commands to get you up and running are shown below,
git clone https://github.com/rickdonato/nornir-napalm-netbox-demo
cd nornir-napalm-netbox-demo/
make add-venv-py3.6
Once complete, your ls -l will look like the below.
.
|-- Makefile
|-- README.md
|-- config.yaml
|-- data
| `-- configs
|-- requirements.txt
|-- scripts
| |-- __init__.py
| |-- backup_configs.py
| |-- helpers.py
| |-- secrets.py
| |-- create_interfaces.py
| `-- update_interfaces.py
`-- venv
Note: For those of you who haven't used Makefiles before they simply allow us to bundle a number of shell commands together and run them via a single make statement. If you want to learn more, feel free to either perform a cat Makefile to see the contents or for a more detailed description of Makefile click here.
To validate Nornir and Napalm are installed check the installed packages via pip freeze, like so:
root@desktop:/tmp/nornir-napalm-netbox-demo# source venv/bin/activate
(venv) root@desktop:/tmp/nornir-napalm-netbox-demo# pip freeze | grep -E "nornir|napalm"
napalm==2.4.0
nornir==2.3.0
Configuration
Creating Devices in Netbox
You will need to create your devices within Netbox. The step-by-step instructions on how to do this is outside the scope of this document but at a high level, you will need to:
- Create Manufacturers under Device Types.
- Create Device Types under Device Types.
- Create Platforms under Devices.
- Create Device Roles under Devices.
- Create Devices under Devices.
- Within each device:
- add a management interface.
- mark management interface as primary.
- assign an IP.
Figure 4 - Adding Devices to Netbox.
Nornir-to-Netbox Configuration
We will now turn our attention to integrating Netbox with Nornir via the Netbox plugin.
First of all create an API Token via the Netbox UI like so,
Figure 5 - Create Netbox API Token.
Next, edit config.yaml (shown below) adding your Netbox API Token and Netbox host details.
---
core:
num_workers: 100
inventory:
plugin: nornir.plugins.inventory.netbox.NBInventory
options:
nb_url: 'http://<NETBOX_HOST>:32768'
nb_token: '<NETBOX_API_TOKEN>'
ssl_verify: False
transform_function: "helpers.adapt_user_password"
Unfortunately, the Nornir Netbox plugin doesn't support Netbox secrets, therefore to inject the required credentials into our inventory we will use a transform function. Our transform function (shown below) will perform a key lookup from scripts/secrets.py for the username and password and then reassign the host.username and host.password inventory attributes.
scripts/helpers.py
from secrets import creds
def adapt_user_password(host):
host.username = creds[f"{host}"]["username"]
host.password = creds[f"{host}"]["password"]
scripts/secrets.py
creds = {
"vqfx": {"username": "admin", "password": "Juniper"},
"vios": {"username": "cisco", "password": "cisco"},
"veos": {"username": "admin", "password": "arista"},
}
Note: Although it goes without saying, having your passwords in a plain text file (for production) is not recommended. Therefore, in a production based environment I would recommend to use a password management tool like Vault. The integration required for this, however, is outside of the scope of this article.
With all the previous configuration complete we can now test that everything is working by printing the inventory, via scripts/helpers.py --inventory:
# scripts/helpers.py --inventory
{'defaults': {'connection_options': {},
'data': {},
'hostname': None,
'password': None,
'platform': None,
'port': None,
'username': None},
'groups': {},
'hosts': {'veos': {'connection_options': {},
'data': {'asset_tag': None,
'model': 'eos',
'role': 'switch',
'serial': '',
'site': 'site1',
'vendor': 'Arista'},
'groups': [],
'hostname': '172.29.133.4',
'password': 'arista',
'platform': 'eos',
'port': None,
'username': 'admin'},
'vios': {'connection_options': {},
'data': { 'asset_tag': None,
'model': 'ios',
'role': 'switch',
'serial': '',
'site': 'site1',
'vendor': 'Cisco'},
'groups': [],
'hostname': '172.29.133.3',
'password': 'cisco',
'platform': 'ios',
'port': None,
'username': 'cisco'},
...
Examples
With the previous configuration complete, let's turn our attention to our examples.
Backup Configurations
Overview
First we will build a script (scripts/backup_configs.py) that will leverage a custom Nornir task to perform a backup of our network device configurations, saving the configs to a local directory.
In the real world this example/script could be extended to run as part on a regular basis (via crontab) or imported as a module. In addition you could also add additional logic to push this up to a code repo such as Github.
In order to achieve this will build a single Python file. Like so:
#!./venv/bin/python
from nornir.plugins.tasks import networking
from nornir.plugins.functions.text import print_result
from nornir.plugins.tasks.files import write_file
from nornir import InitNornir
BACKUP_PATH = "./data/configs"
def backup_config(task, path):
r = task.run(task=networking.napalm_get, getters=["config"])
task.run(
task=write_file,
content=r.result["config"]["running"],
filename=f"{path}/{task.host}.txt",
)
nr = InitNornir(config_file="./config.yaml")
devices = nr.filter(role="switch")
result = devices.run(
name="Backup Device Configurations", path=BACKUP_PATH, task=backup_config
)
print_result(result, vars=["stdout"])
Code Explanation
Let us go through the script on some more detail.
First we perform our imports and set the location where we want to save our configurations to.
from nornir.plugins.tasks import networking
from nornir.plugins.functions.text import print_result
from nornir.plugins.tasks.files import write_file
from nornir import InitNornir
BACKUP_PATH = "./data/configs"
Next we create a task. This task performs 2 main actions:
- Napalm Getter - Using the
napalmplugin it runs theconfig(akaget_config) Napalm getter. - Write File - The
write_fileplugin saves the result into our intended destination.
def backup_config(task, path):
r = task.run(task=networking.napalm_get, getters=["config"])
task.run(
task=write_file,
content=r.result["config"]["running"],
filename=f"{path}/{task.host}.txt",
)
We initialise Nornir against the configuration file and then filter the devices we are interested in (i.e we will later run the task against). Remember the role selected in the filter is the role defined/assigned within Netbox.
nr = InitNornir(config_file="./config.yaml")
devices = nr.filter(role="switch")
Finally we bring it all together. We run our task backup_config against our devices object, also passing in any variables that the task requires. In our case BACKUP_PATH.
Once done, we print the results out to screen.
result = devices.run(
name="Backup Device Configurations", path=BACKUP_PATH, task=backup_config
)
print_result(result, vars=["stdout"])
Example
Updating Netbox
Overview
Our next example will be based on populating Netbox using data collected from our devices.
This will be a little bit more involved, but in essence is based on 2 steps:
- Create Interface -
scripts/create_interfaces.py- Pulling the interface data from the device and creating the related interfaces. - Update Interface -
scripts/update_interfaces.py- Pulling the interface data from the device and updating the corresponding interfaces in Netbox.
To update Netbox we will use python-netbox (read more). Let’s look at the code:
scripts/create_interface.py
import re
from nornir.plugins.tasks import networking
from nornir.plugins.functions.text import print_result
from nornir import InitNornir
from netbox import NetBox
from helpers import get_device_id, is_interface_present
import pprint
nr = InitNornir(config_file="./config.yaml")
nb_url, nb_token, ssl_verify = nr.config.inventory.options.values()
nb_host = re.sub("^.*//|:.*$", "", nb_url)
netbox = NetBox(host=nb_host, port=32768, use_ssl=False, auth_token=nb_token)
nb_interfaces = netbox.dcim.get_interfaces()
def create_netbox_interface(task, nb_interfaces, netbox):
r = task.run(task=networking.napalm_get, getters=["interfaces"])
interfaces = r.result["interfaces"]
for interface_name in interfaces.keys():
if not is_interface_present(nb_interfaces, f"{task.host}", interface_name):
print(
f"* Creating Netbox Interface for device {task.host}, interface {interface_name}"
)
device_id = get_device_id(f"{task.host}", netbox)
netbox.dcim.create_interface(
name=f"{interface_name}",
form_factor=1200, # default
device_id=device_id,
)
devices = nr.filter(role="switch")
result = devices.run(
name="Create Netbox Interfaces",
nb_interfaces=nb_interfaces,
netbox=netbox,
task=create_netbox_interface,
)
print_result(result, vars=["stdout"])
scripts/update_interface.py
import re
from nornir.plugins.tasks import networking
from nornir.plugins.functions.text import print_result
from nornir import InitNornir
from netbox import NetBox
from helpers import is_interface_present
nr = InitNornir(config_file="./config.yaml")
nb_url, nb_token, ssl_verify = nr.config.inventory.options.values()
nb_host = re.sub("^.*//|:.*$", "", nb_url)
netbox = NetBox(host=nb_host, port=32768, use_ssl=False, auth_token=nb_token)
nb_interfaces = netbox.dcim.get_interfaces()
def update_netbox_interface(task, nb_interfaces):
r = task.run(task=networking.napalm_get, getters=["interfaces"])
interfaces = r.result["interfaces"]
for interface_name in interfaces.keys():
mac_address = interfaces[interface_name]["mac_address"]
if mac_address == "None" or mac_address == "Unspecified":
mac_address = "ee:ee:ee:ee:ee:ee"
description = interfaces[interface_name]["description"]
if is_interface_present(nb_interfaces, f"{task.host}", interface_name):
print(
f"* Updating Netbox Interface for device {task.host}, interface {interface_name}"
)
netbox.dcim.update_interface(
device=f"{task.host}",
interface=interface_name,
description=description,
mac_address=mac_address,
)
devices = nr.filter(role="switch")
result = devices.run(
name="Update Netbox Interfaces",
nb_interfaces=nb_interfaces,
task=update_netbox_interface,
)
print_result(result, vars=["stdout"])
Code Explanation
Within both scripts once we perform the initial imports, we pull the Netbox config details from the Nornir inventory which we then pass in at the point we instantiate netbox.
nb_url, nb_token, ssl_verify = nr.config.inventory.options.values() \
nb_host = re.sub("^.*//|:.*$", "", nb_url) \
\
netbox = NetBox(host=nb_host, port=32768, use_ssl=False, auth_token=nb_token)
The overall layout is the same as the previous backup configuration example. i.e a task is created, we filter the devices we require and then run our task against the devices, passing in our variables.
In terms of the task logic they are somewhat similar:
scripts/create_interface.py
- We pull the interfaces from Netbox via
netbox.dcim.get_interfaces(). - We iterate over the interface
dict()keys that are returned. - For each interface we check to see if the interface is already in Netbox via our helper
is_interface_present. - If the interface is not present we perform a
netbox.dcim.create_interface().
scripts/update_interface.py
- We pull the interfaces from Netbox via
netbox.dcim.get_interfaces(). - We iterate over the interface dict() keys that are returned.
- For each interface we check to see if the interface is already in Netbox via our helper
is_interface_present. - If the interface is present we perform a
netbox.dcim.update_interface(). - The data we populate is the description and MAC address.
Example
Figure 6 - Populated Netbox data.
Outro
That concludes this series around Nornir, Napalm and Netbox. I can honestly say this article has been a blast to create. As you can see these tools, give you enormous potential and flexibility into how you automate your network. Abstracting many of the lower level details that would typically take a lot of time and patience to develop.
Thanks for reading, and remember to check out our free newsletter to get all the latest PacketFlow updates.
References
"NetBox - Read the Docs." https://netbox.readthedocs.io/. Accessed 23 Oct. 2019. ↩︎
"nornir-automation/nornir: Pluggable multi-threaded ... - GitHub." https://github.com/nornir-automation/nornir. Accessed 23 Oct. 2019. ↩︎
"Python Virtual Environment | Introduction - GeeksforGeeks." https://www.geeksforgeeks.org/python-virtual-environment/. Accessed 26 Oct. 2019. ↩︎
Comments