Device Profiles¶
Most network equipment vendors use different syslog message format to each other,
some even use a different format for each of their devices. For napalm-logs
to be able to take a syslog message from a device and output it in a
vendor-agnostic way, it needs to know the format of that device’s messages.
Each network operating system has a set of profiles, defined under a directory
with the name of the platform, by default defined under napalm_logs/config
.
For example, the profiles for eos
are defined under
napalm_logs/config/eos/
, for junos
under napalm_logs/config/junos/
and so on.
Directory tree structure example:
napalm_logs/config/
├── __init__.py
├── eos
│ └── init.yml
├── iosxr
│ └── __init__.py
├── junos
│ └── init.yml
└── nxos
└── init.yml
The user can select to extend the capabilities of the public library, by defining profiles under a directory, specifying the path using the extension-config-path option.
Custom directory tree example:
/pat/to/custom/config/
├── eos
│ └── bgp_3_notification.py
├── junos
└── init.yml
└── UI_DBASE_LOGIN_EVENT.py
└── SNMP_TRAP_LINK_DOWN.py
Each syslog message can be divided into two logical sections:
- the identification section, which provides enough information to identify the operating system that generated the message, together with other details, such as datetime, hostname, PRI, process daemon, PID, etc. In napalm-logs, this section will be referenced as prefix.
- the actual message section, which is the part of the syslog message which contains the useful information. In napalm-logs, this section will be referenced as message.
Example: given the message Mar 30 12:45:19 re0.edge01.bjm01 rpd[15852]: BGP_PREFIX_THRESH_EXCEEDED 1.2.3.4 (External AS 15169): Configured maximum prefix-limit threshold(160) exceeded for inet-unicast nlri: 181 (instance master)
:
Mar 30 12:45:19 re0.edge01.bjm01 rpd[15852]: BGP_PREFIX_THRESH_EXCEEDED
is the prefix section.1.2.3.4 (External AS 15169): Configured maximum prefix-limit threshold(160) exceeded for inet-unicast nlri: 181 (instance master)
is the message section.
Both sections are platform-specific, and the prefix part can be used to idenfiy the operating system that generated a certain syslog message. The identification is done via prefix matchers (prefix parsers). Similarly, the extraction of the information from the message section is done via message parsers.
Please note that some platforms do not respect a single prefix pattern, but a variety, this is why we need a couple of prefix matchers.
YAML Profiles¶
Each config file has two distinct sections, one to identify the OS that the
message originated from (called prefixes
), and one to identify each log
message that napalm-logs
should convert (called messages
).
prefixes
¶
This section defines what we have defined above as prefix matches, or prefix parsers, for the OS in question.
Here is the config for junos
:
prefixes:
- time_format: "%b %d %H:%M:%S"
values:
date: (\w+\s+\d+)
time: (\d\d:\d\d:\d\d)
hostPrefix: (re\d.)?
host: ([^ ]+)
processName: /?(\w+)
processId: \[?(\d+)?\]?
tag: (\w+)
line: '{date} {time} {hostPrefix}{host} {processName}{processId}: {tag}: '
Note
Prefix parsers are usually defined as __init__.yml
, init.yml
or
index.yml
.
What does each option mean?
line
¶
This represents the format of the part of the log message that present most of
the time. Each section of the message that can change should be replaced by a
variable. If a variable isn’t always present then you should add it to the line
but make that variable optional (covered in the values
section).
Any white space in line
will match any number of contiguous white space,
therefore if it is possible for there to be either one white space or two white
spaces, you should only add one white space to line
.
values
¶
This is used to specify the regex pattern for each of the variables specified
in line
. All variables in line
should have an entry under values
,
even if you have no use for them.
Each of these variables will be output in a message dict after processing.
messages
¶
Here is where all log messages that should be matched are specified.
Note
Message parsers are usually defined under a YAML file having the name of the error ID they produce. However, this is not absolutely mandatory.
Here is an example message:
messages:
- error: INTERFACE_DOWN
tag: SNMP_TRAP_LINK_DOWN
values:
snmpID: (\d+)
adminStatusString|uppercase: (\w+)
adminStatusValue: (\d)
operStatusString|uppercase: (\w+)
operStatusValue: (\d)
interface: ([\w\-\/]+)
line: 'ifIndex {snmpID}, ifAdminStatus {adminStatusString}({adminStatusValue}), ifOperStatus {operStatusString}({operStatusValue}), ifName {interface}'
model: openconfig_interfaces
mapping:
variables:
interfaces//interface//{interface}//state//admin_status: adminStatusString
interfaces//interface//{interface}//state//oper_status: operStatusString
static: {}
What does each option mean?
error
¶
This is the vendor agnostic ID for the error message, the error
for each
message should be unique. Currently we are using the junos
definitions where
possible, this is likely to change.
tag
¶
This is the unique ID from the device itself.
This field is used when identifying if the log message is related to the configured error. Some devices use the same name for different types of logs, therefore this does not need to be unique.
If you look at the config for prefix
above, you will see the variable
tag
in line
, this is the same tag
as configured here and matched on.
match_on
: tag
¶
This field name the field that try to match on. Defaults to tag
.
line
¶
This is the same as line
above.
values
¶
This is the same as values
above, other than the fact they can be used in
mapping
(this will be covered under mapping
). You can manipulate these
values using replace functions found in napalm_logs.utils.Replace i.e
adminStatusString|uppercase
.
model
¶
This is the YANG model to use to output the log message. You can find all models and their structure here.
mapping
¶
This shows where in the OpenConfig model each of the variables in the message
should be placed. There are two options, variables
and static
.
variables
should be used when the value being set is taken from the message,
and static
should be used when the value is manually set.
state
¶
state
is an optional config bit which may be useful when defining messages
that have a counter-part. For example: MAJOR_ALARM_SET
/ MAJOR_ALARM_CLEARED
. The idea behind this is to have pairs or groups of
notifications that have a specific significance mapping out to a desired value
(e.g., the state value for MAJOR_ALARM_SET
can be 1, while the state value
for MAJOR_ALARM_CLEARED
can be 0). The value can be any number, not only
binary, when the group is larger than two notifications.
It is equally important to not that when using the Prometheus Publisher, when using this field, an additional metric is being exposed, providing the state value, besides the usual counter.
The metric name is derived from the base name of the notification, by stripping
the last part (after underscore) and replacing it with _state
. For
instance, continuing the example above, MAJOR_ALARM_SET
and
MAJOR_ALARM_CLEARED
would both set the same Gauge metric
napalm_logs_major_alarm_state
. If the notification groups don’t have
a common base name for whatever reason, you can define individual state tags
using the state_tag
option (see below).
state_tag
¶
This option provides a custom name for the state metric on groups of notifications. By default, this is not necessary, as it’s assumed a group has the same base name, but it may not always be the case.
Based on the example above, MAJOR_ALARM_SET
and MAJOR_ALARM_CLEARED
would, by default, set the napalm_logs_major_alarm_state
Gauge metric,
however, by providing the value state_tag: system_alarm_state
(for both), the
metric becomes: napalm_logs_system_alarm_state
.
Pure Python profiles¶
Writing YAML profiles is flexible and fast, but this model comes with many
logical limitations. For this reason, the developer can equally write pure
Python prefixes
or messages
parsers. They can be defined under the same
directory as the YAML descriptors, and they will be loaded dynamically.
Note
The user is allowed to use any combination of YAML and pure Python parsers to match the messages and defined the prefixes.
Similarly to the YAML profilers, the Python profiles have two logical sections:
prefixes
that provide the operating system identification and messages
that extract the information from the raw syslog messages and maps to an
object having the YANG hierarchy. Both are free-form Python modules,
with a single constraint that will be explained below.
prefixes
¶
A pure Python module that provides the prefix configuration, in order to identify the operating system generating the message.
A module providing the prefix needs to define a function called extract
that takes a single argument, msg
which is the raw syslog message received
from the network device. The function has to return a dictionary with the
parts extracted from the syslog message, without any further processing. The
following keys are mandatory:
host
: the network device hostname, as provided in the syslog message prefix section.tag
: which is the unique identification tag of the syslog message, e.g. in the messageMar 30 12:45:19 re0.edge01.bjm01 rpd[15852]: BGP_PREFIX_THRESH_EXCEEDED 1.2.3.4 (External AS 15169): Configured maximum prefix-limit threshold(160) exceeded for inet-unicast nlri: 181 (instance master)
, thetag
isBGP_PREFIX_THRESH_EXCEEDED
. Other tag examples:bgp_read_message
,ROUTING-BGP-5-MAXPFX
or evenAlarm set
.message
: is the message that what we have defied earlier as the message section, e.g.User 'dummy' entering configuration mode
.
Note
Prefix parsers are usually defined as __init__.py
, init.py
or
index.py
.
The following example is a Python prefix parser for NX-OS:
import re
from collections import OrderedDict
import napalm_logs.utils
_RGX_PARTS = [
('pri', r'(\d+)'),
('host', r'([^ ]+)'),
('date', r'(\d+ \w+ +\d+)'),
('time', r'(\d\d:\d\d:\d\d)'),
('timeZone', r'(\w\w\w)'),
('tag', r'([\w\d-]+)'),
('message', r'(.*)')
]
_RGX_PARTS = OrderedDict(_RGX_PARTS)
_RGX = '\<{0[pri]}\>{0[host]}: {0[date]} {0[time]} {0[timeZone]}: %{0[tag]}: {0[message]}'.format(_RGX_PARTS)
def extract(msg):
return napalm_logs.utils.extract(_RGX, msg, _RGX_PARTS)
The example above matches messages from NX-OS looking like: <190>sw01.bjm01: 2017 Jul 26 14:42:46 UTC: %SOME-TAG: this is a very useful syslog message
,
and extracts the following details:
pri
: 190host
: sw01.bjm01tag
: SOME-TAGdate
: 2017 Jul 26time
: 14:42:46timeZone
: UTCmessage
: this is a very useful syslog message
These details are returned by the extract
function, which returns a
dictionary such as:
{
'pri': '190',
'host': 'sw01.bjm01',
'tag': 'SOME-TAG',
'time': '14:42:46',
'date': '2017 Jul 26',
'timeZone': 'UTC',
'message': 'this is a very useful syslog message'
}
Except tag
, host
and message
, all the other fields can be optional,
and they are platform-specific (or even message-type-specific, in some very
sad cases). However, there are some particular cases when the other fields can
provide interesting information, eventually to be used to match messages using
the match_on
option.
messages
¶
Writing a message parser can be equally simple and flexible, the rules to consider being:
- Define a function called
emit
that generates the syslog message. - A dunder called
__yang_model__
that specifies the YANG model. - A variable names
__tag__
that specifies the tag name, that is used to match when comparing the value of thetag
field extracted from the message prefix and determine what parser should process the syslog message. However, this variable is optional – when not defined, it will use the filename as tag. - A variable called
__error__
that defines the name of the global error. Each structured message published by napalm-logs has a certain error tag, that is unique and cross-platform. This variable is also optional – when not defined, the error ID will be the file name.
Note
Message parsers are usually defined under a Python file having the name of the error ID they produce. However, this is not absolutely mandatory.
Useful functions¶
At times, the developer may find very useful several functions, in order to acomplish recurrent tasks:
napalm_logs.utils.extract
: Extracts the fields from a unstructured text, given a field-regex mapping. Please check the previous paragraph for an usage example.napalm_logs.utils.setval
: Set a value under the dictionary hierarchy identified under the key. The key'foo//bar//baz'
will configure the value under the dictionary hierarchy{'foo': {'bar': {'baz': {}}}}
. Example:
>>> napalm_logs.utils.setval('foo//bar//baz', 'value')
{'foo': {'bar': {'baz': 'value'}}}
napalm_logs.utils.traverse
: Traverse a dict or list using a slash delimiter target string. The target'foo//bar//0'
will returndata['foo']['bar'][0]
if this value exists, otherwise will return empty dict. ReturnNone
when not found. This can be used to verify if a certain key exists under dictionary hierarchy.