Napalm-logs¶
Python library to parse syslog messages from network devices and produce JSON serializable Python objects, in a vendor agnostic shape. The output objects are structured following the OpenConfig or IETF YANG models.
For example, the following syslog message from a Juniper device:
<149>Jun 21 14:03:12 vmx01 rpd[2902]: BGP_PREFIX_THRESH_EXCEEDED: 192.168.140.254 (External AS 4230): Configured maximum prefix-limit threshold(140) exceeded for inet4-unicast nlri: 141 (instance master)
Will produce the following object:
{
"yang_message": {
"bgp": {
"neighbors": {
"neighbor": {
"192.168.140.254": {
"state": {
"peer_as": "4230"
},
"afi_safis": {
"afi_safi": {
"inet4": {
"state": {
"prefixes": {
"received": 141
}
},
"ipv4_unicast": {
"prefix_limit": {
"state": {
"max_prefixes": 140
}
}
}
}
}
}
}
}
}
}
},
"message_details": {
"processId": "2902",
"severity": 5,
"facility": 18,
"hostPrefix": null,
"pri": "149",
"processName": "rpd",
"host": "vmx01",
"tag": "BGP_PREFIX_THRESH_EXCEEDED",
"time": "14:03:12",
"date": "Jun 21",
"message": "192.168.140.254 (External AS 4230): Configured maximum prefix-limit threshold(140) exceeded for inet4-unicast nlri: 141 (instance master)"
},
"timestamp": 1498053792,
"facility": 18,
"ip": "127.0.0.1",
"host": "vmx01",
"yang_model": "openconfig-bgp",
"error": "BGP_PREFIX_THRESH_EXCEEDED",
"os": "junos",
"severity": 5
}
The library is provided with a command line program which acts as a daemon, running in background and listening to syslog messages continuously, then publishing them over secured channels, where multiple clients can subscribe.
It is flexible to listen to the syslog messages via UDP or TCP, but also from brokers such as Apache Kafka. Similarly, the output objects can be published via various channels such as ZeroMQ, Kafka, or remote server logging. It is also pluggable enough to extend these capabilities and listen or publish to other services, depending on the needs.
The messages are published over a secured channel, encrypted and signed. Although the security can be disabled, this is highly discouraged.
Output data¶
The objects published by napalm-logs are structured data, with the hierarchy standardized in the OpenConfig and IETF models. To check what models are used for each message type, together with examples of raw syslog messages and sample output objects, please check the Structured Messages section.
Install¶
napalm-logs is available on PyPi and can easily be installed using the following command:
$ pip install napalm-logs
For advanced installation notes, see Installation.
How to use napalm-logs¶
Basic Configuration¶
Firstly you need to decide if you would like all messages between napalm-logs and the clients to be encrypted. If you do want them to be encrypted you will require a certificate and key, which you can generate using the following command:
openssl req -nodes -x509 -newkey rsa:4096 -keyout /var/cache/napalm-logs.key -out /var/cache/napalm-logs.crt -days 365
This will provide a self-signed certificate napalm-logs.crt
and key
napalm-logs.key
under the /var/cache
directory.
If you do not require the messages to be encrypted you can ignore the above
step and just use the command line argument --disable-security
when starting
napalm-logs.
Each of the other config options come with defaults, so you can now start napalm-logs with default options and your chosen security options.
Starting napalm-logs¶
Napalm-logs will need to be run with root privileges if you want it to be able
to listen on udp
port 514
- the standard syslog port. If you need to
run it via sudo and it has been installed in a virtual env, you will need to
include the full path. In these examples I will run as root.
To start napalm-logs using the crt and key generated above you should run the following command:
napalm-logs --certificate /var/cache/napalm-logs.crt --keyfile /var/cache/napalm-logs.key
This will start napalm-logs listening for incoming syslog messages on
0.0.0.0
port 514
. It will also start to listen for incoming client
requests on 0.0.0.0
port 49017
, and incoming authentication requests on
0.0.0.0
port 49018
.
For more information on authentication please see the Client Authentication
section.
Further Configuration¶
It is possible to change the address and ports that napalm-logs will use, let’s take a look at these options:
-a ADDRESS, --address=ADDRESS
Listener address. Default: 0.0.0.0
-p PORT, --port=PORT Listener bind port. Default: 514
--publish-address=PUBLISH_ADDRESS
Publisher bind address. Default: 0.0.0.0
--publish-port=PUBLISH_PORT
Publisher bind port. Default: 49017
--auth-address=AUTH_ADDRESS
Authenticator bind address. Default: 0.0.0.0
--auth-port=AUTH_PORT
Authenticator bind port. Default: 49018
There are several plugable
parts to napalm-logs, two of which are the
listener
and the publisher
. The listener is the part that ingests the
incoming syslog messages, and the publisher is the part that outputs them to the
client.
You can chose which listener to use, and which publisher to use by using the following arguments:
--listener=LISTENER Listener type. Default: udp
-t TRANSPORT, --transport=TRANSPORT
Publish transport. Default: zmq
There are more configuration options, please see Configuration Options for more details.
Configuration file example¶
The napalm-logs server can be started without any CLI aguments, as long as they
are correctly specified under the configuration file. The default path of the
configuration file is under /etc/napalm/logs
. To select a different
filepath, we can use the -c
option:
napalm-logs -c /home/admin/napalm/logs
The configuration file is formatted as YAML, which makes it more human readable.
In general, any configuration option available on the CLI can be specified in
the configuration file, with the mention that hyphen is replaced by underscore,
e.g.: the CLI option auth-address
becomes auth_address
in the
napalm-logs configuration file.
address: 172.17.17.1
port: 5514
publish_address: 172.17.17.2
publish_port: 49017
transport: zmq
listener:
kafka:
bootstrap_servers:
- 10.10.10.1
- 10.10.10.2
- 10.10.10.3
The configuration above listens to the syslog messages from the Kafka bootstrap
servers 10.10.10.1
, 10.10.10.2
and 10.10.10.3
then publishes the
structured objects encrypted and serialized via ZeroMQ, serving them at the
address 172.17.17.2
, port 49017
.
Check the complete list of configuration options under Configuration Options.
Starting a Client¶
The client structure depends on how you start the napalm-logs daemon. If the
security is disabled (via the CLI option --disable-security
or through the
configuration file, where the disable_security
field is set as false
),
the client script is as simple as:
#!/usr/bin/env python
import zmq
import napalm_logs.utils
server_address = '127.0.0.1'
server_port = 49017
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://{address}:{port}'.format(address=server_address,
port=server_port))
socket.setsockopt(zmq.SUBSCRIBE, '')
while True:
raw_object = socket.recv()
print(napalm_logs.utils.unserialize(raw_object))
Which subscribes to the ZeroMQ bus and deserializes messages using the
napalm_logs.utils.unserialise
helper. The server_address
and the
server_port
of the client represent the --publish-address
and the
--publish-port
of the napalm-logs daemon.
When the program is started with security enabled (recommended), the
clients can use the napalm_logs.utils.ClientAuth
class, which executes the
handshake to retrieve the encryption key and hex of the verification key. This
class requires the certificate (the same certificate specified when starting
the napalm-logs daemon), as well as the authentication address and port
(corresponding to the --auth-address
and --auth-port
CLI arguments or
auth_address
and auth_port
configuration fields sent to the napalm-logs
daemon):
#!/usr/bin/env python
import napalm_logs.utils
import zmq
server_address = '127.0.0.1'
server_port = 49017
auth_address = '127.0.0.1'
auth_port = 49018
certificate = '/var/cache/napalm-logs.crt' # This is the server crt generated earlier
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect('tcp://{address}:{port}'.format(address=server_address,
port=server_port))
socket.setsockopt(zmq.SUBSCRIBE, '')
auth = napalm_logs.utils.ClientAuth(certificate,
address=auth_address,
port=auth_port)
while True:
raw_object = socket.recv()
decrypted = auth.decrypt(raw_object)
print(decrypted)