Network Automation 101 - Skills and Technologies
Scroll Down
Introduction
Besides understanding and learning about the different tools out there, it is important that the Network Engineer understands key concepts and technologies needed to operate the tools presented before them.
Figure 1: Skills and Technology Overview
Programming and Scripting Languages
Even though you DON’T need to be a Software Developer, it’s important to have some knowledge in programming languages, such as Python or Golang for example.
Python
Python is an Object Oriented Programming (OOP) language popular in the network automation space. Tools like Ansible, Salt, NAPALM, Nornir, Netmiko and pyATS are written with Python.
It is also worth mentioning the usage and operation of template engines, such as the Jinja2 templating language are widely used by configuration tools like Ansible, SaltStack and Nornir.
Some tools discussed in the previous guides are Python only, like Pytest and NAPALM. Therefore, if you want to test them out some python knowledge is required. Here are some other libraries that are used in the network automation space to interact with devices as well:
- juniper/py-junos-eznc - Python library for Junos automation
- arista-eosplus/pyeapi - Python client for Arista eAPI
- Requests - HTTP for Humans
Golang
The Go Programming Language is an open-source programming environment that makes it easy to build reliable and efficient software.
Some of the key benefits of language like Go in this space are:
- One binary.
- Fast compilation.
- Testing, benchmarking and profiling built-in.
- Good performance and concurrency mechanism.
You will find that in Golang there aren't that many packages targeted for the network automation community, but that list is growing.
Data Formats
These are data formats that are used and digested by applications and systems. Commonly used as configuration files, communication between systems and general file data ingestion and are heavily used in the tool landscape shown before. The most common are:
YAML (YAML Ain't Markup Language)
YAML is a data serialization language mostly used in configuration files and playbooks definitions. YAML’s minimal syntax makes it easy to read (except when using complex data structures) and so it’s characterised ‘human readable’.
An example of a YAML file is shown below:
---
vlans:
vlan:
- vlan-id: '111'
config:
vlan-id: 111
name: SALES
status: ACTIVE
- vlan-id: '222'
config:
vlan-id: 222
name: HR
status: ACTIVE
- vlan-id: '333'
config:
vlan-id: 333
name: MARKETING
status: ACTIVE
XML (Extensible Markup Language)
XML is a markup language that defines a set of rules for encoding documents and representing data structures, primarily designed to store and transport data. It is both human and machine readable, but is seen far more in API transactions between systems.
An example of a JunOS router XML RPC call is shown below:
<rpc-reply xmlns:junos="http://xml.juniper.net/junos/14.1R1/junos”>
<rpc>
<get-interface-information>
<interface-name>ge-0/0/0</interface-name>
</get-interface-information>
</rpc>
</rpc-reply>
XML documents form a tree structure that starts at the root and branches to the leaves, so XML documents are formed as element trees.
All elements can have content and attributes, for example on the document above the element interface-name has a content of ge-/0/0/0, and the top element rpc-reply has an attribute of xmlns:junos="http://xml.juniper.net/junos/14.1R1/junos".
JSON (JavaScript Object Notation)
JSON is a data interchange format that uses human readable text to transmit data objects consisting of attribute and value pairs. It was derived from JavaScript but is language-independent. JSON is typically preferred over XML due to JSON`s lightweight nature (i.e there is no requirement for closing tags like XML).
Example of interface definition variables in JSON format:
{
"interfaces": [
{
"name": "Vlan177",
"enabled": true,
"address": "10.77.1.68/28",
"description": "Lan In-Band Network",
"load_interval": 5
},
{
"name": "Management1",
"description": "lab01 - Eth100/1/37",
"enabled": true,
"instance": "MANAGEMENT",
"address": "10.17.17.177/23",
"load_interval": 5
}
]
}
Data Modeling Languages
A data model is an abstraction that organizes elements of data and standardises how they relate to one another.[1]
In the network infrastructure space we have the representation of data in multiple formats (XML, JSON, raw text and so on). The data model can vary depending on the vendor or implementation where the data is collected from. For example, vendor X may provide the VLAN ID under a key of vlan_id via JSON where as vendor Y may provide the VLAN ID against a key of vlanid.
This can create challenges when applications and scripts want to consume the data, since the specific data types and general structure are not always known and uniform. This is where a data model language comes into play.
SMI (Structure of Management Information)
The most common form of data model is the SMI (Structure of Management Information) seen in the SNMP implementation. They describe the structure of the management data and a device subsystem, in which they use a hierarchical namespace containing Object Identifiers (OIDs), these hierarchies are described as a Management Information Base (MIB).
SNMP didn’t actually take off as a widely used technology for network device configuration deployment. One of the key reasons for this is that it cannot discern configurational from operational state and statistical data.
YANG (Yet Another Next Generation)
YANG came into the picture as a data modeling language to address some of the current drawbacks, support API contracts and to be used with newer management protocols that supported transaction-based communications (NETCONF and RESTCONF).
This data modeling language can be used to model both configuration as well as operational state of network elements and also defines the format of the event notifications emitted by network elements. It is protocol independent and can be converted into any encoding format (XML, JSON, etc.).
Below is an ASCII representation of an OpenConfig LACP YANG model:
pyang -f tree oc-models/lacp/openconfig-lacp.yang -p oc-models/
module: openconfig-lacp
+--rw lacp
+--rw config
| +--rw system-priority? uint16
+--ro state
| +--ro system-priority? uint16
+--rw interfaces
+--rw interface* [name]
+--rw name -> ../config/name
+--rw config
| +--rw name? oc-if:base-interface-ref
| +--rw interval? lacp-period-type
| +--rw lacp-mode? lacp-activity-type
| +--rw system-id-mac? oc-yang:mac-address
| +--rw system-priority? uint16
+--ro state
| +--ro name? oc-if:base-interface-ref
| +--ro interval? lacp-period-type
| +--ro lacp-mode? lacp-activity-type
| +--ro system-id-mac? oc-yang:mac-address
| +--ro system-priority? uint16
+--ro members
+--ro member* [interface]
+--ro interface -> ../state/interface
+--ro state
+--ro interface? oc-if:base-interface-ref
+--ro activity? lacp-activity-type
+--ro timeout? lacp-timeout-type
+--ro synchronization? lacp-synchronization-type
+--ro aggregatable? boolean
+--ro collecting? boolean
+--ro distributing? boolean
+--ro system-id? oc-yang:mac-address
+--ro oper-key? uint16
+--ro partner-id? oc-yang:mac-address
+--ro partner-key? uint16
+--ro port-num? uint16
+--ro partner-port-num? uint16
+--ro counters
+--ro lacp-in-pkts? oc-yang:counter64
+--ro lacp-out-pkts? oc-yang:counter64
+--ro lacp-rx-errors? oc-yang:counter64
+--ro lacp-tx-errors? oc-yang:counter64
+--ro lacp-unknown-errors? oc-yang:counter64
+--ro lacp-errors? oc-yang:counter64
The actual YANG model for the above can be seen here: https://github.com/openconfig/public/blob/master/release/models/lacp/openconfig-lacp.yang
Application Programming Interfaces (APIs)
The data structures, serializations and models help define what we manipulate from network devices and systems. The APIs help us define how we get and manipulate that data.
An API is a mechanism that allows the interaction between applications using a set of specifications and protocols. Functioning as an interface for systems, it makes them communicate with each other like chatops, messaging systems, alerts/events systems, NoSQL databases, Network control plane and data plane, and so on.
APIs are mainly defined by the transport protocol, data specification and rules bound to it. Here are the main ones:
RESTful APIs
RESTful-API is a representational state transfer API that defines a set of constraints to be used when interacting with web services. HTTP normally handles the transport for the data payload which typically uses a data formatting language of HTML, XML, or JSON. REST-based APIs support CRUD based operations - Create, Read, Update and Delete. These operations are performed via the following HTTP methods:
| HTTP Methods | Action |
|---|---|
| GET | Obtain information about a resource. |
| POST | Create a new resource. |
| PUT | Update a resource. |
| DELETE | Delete a resource. |
Below shows a small example. Here we perform an HTTP GET against a public API to obtain our IP address and location. Notice how the response is JSON based.
$ curl -X GET -s https://api.myip.com | jq
{
"ip": "11.228.32.167",
"country": "France",
"cc": "FR"
}
NETCONF
NETCONF is used mainly on network devices to install, manipulate and delete configuration data of the device. Its operations are done using a Remote Procedure Call (RPC) via SSH transport using XML (or JSON) based data encoding for YANG data models.
Some of the key features to NETCONF are the ability to rollback configurations, the ability to support any data model and the separation of config from operational state.
Figure 2: NETCONF Stack
RESTCONF
“an HTTP-based protocol that provides a programmatic interface for accessing data defined in YANG, using the datastore concepts defined in the Network Configuration Protocol (NETCONF).” - RFC8040.
Similarly to NETCONF, it works as an HTTP-based protocol to provide a programmatic interface for accessing data defined in YANG and supports data encoding of either JSON or XML.
Figure 3: RESTCONF Stack
gNMI and gRPC
Coming from Google and the OpenConfig consortium, the gNMI specification is a framework that stands for gRPC Network Management Interface (where gRPC stands for gRPC Remote Procedure Call).
From a protocol perspective it resembles NETCONF and RESTCONF, and it can use YANG data modeling as the interface description language.
gRPC is a mechanism that allows a client to invoke operations on the server and, together with the gNMI specification, it defines a particular set of gRPC operations that correspond more or less to the NETCONF/RESTCONF operations and subscription mechanism.
Containers and Docker
Development and testing in containers is becoming more popular and is used in different types of scenarios. From testing a python application to serving as a workstation in a virtual environment.
Containers
What is a container? Unlike a VM (virtual machine), which is a virtualized instance of an OS, a container is a logically isolated slice of an operating system, i.e. unlike a VM each container shares a kernel with other containers and other system libraries. This makes containers extremely lightweight and fast to run.
Figure 4: Containers vs VM Overview[2]
Docker
Docker is a containerisation tool which is now used as a synonymous term when talking about containers thanks to its popularity. It has become the go-to tool for developers, systems and network infrastructure engineers. It provides portability, ease of testing and is a lightweight option compared to VMs.
Most network simulation tools now support it (GNS3, EVE-NG and so on), and they help create a more dynamic and interesting environment for your network topologies since you can easily spin up a Elasticsearch, Logstash and even an SFTP server container.
References
"Data model - Wikipedia." https://en.wikipedia.org/wiki/Data_model. Accessed 8 Jan. 2020. ↩︎
"Containers vs. Virtual Machines (VMs) - NetApp Blog." 16 Mar. 2018, https://blog.netapp.com/blogs/containers-vs-vms/. Accessed 19 Feb. 2020. ↩︎