Skip to content

Commit 568d37e

Browse files
committed
Merge remote-tracking branch 'origin/develop' into scetron-mroutes-and-igmp
2 parents cab189d + 091d784 commit 568d37e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

84 files changed

+44029
-41047
lines changed

.pylintrc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -763,5 +763,5 @@ min-public-methods=2
763763

764764
# Exceptions that will emit a warning when being caught. Defaults to
765765
# "BaseException, Exception".
766-
overgeneral-exceptions=BaseException,
767-
Exception
766+
overgeneral-exceptions=builtins.BaseException,
767+
builtins.Exception

build/requirements.txt

Lines changed: 106 additions & 92 deletions
Large diffs are not rendered by default.

docs/config_file.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,8 @@ coalescer:
4343
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |:-------------------------------- | ------------------- |
4444
| data-directory | The directory where the poller stores the data collected from the network | - | yes |
4545
| temp-directory | Where SuzieQ stores temporary data | /tmp/.suzieq/ | no |
46-
| rest.rest_certfile | certificate for the REST server | - | no |
47-
| rest.rest_keyfile | keyfile for the REST server | - | no |
46+
| rest.rest-certfile | certificate for the REST server | - | no |
47+
| rest.rest-keyfile | keyfile for the REST server | - | no |
4848
| rest.API_KEY | API key for the REST server | - | yes (if using rest) |
4949
| rest.address | IP address of the REST server. | 127.0.0.1 | no |
5050
| rest.port | port of the REST server | 80 | no |

docs/gui.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,121 @@ Some additional arguments are allowed to set up the GUI:
1515

1616
- `-p --port`: defines in which port the GUI is going to run. By default is `8501`
1717
- `-c --config`: SuzieQ configuration file to use. Check the [Configuration](config_file.md) page to check the default configuration file location.
18+
19+
## Start a GUI behind an nginx reverse proxy
20+
21+
This section is an step by step guide on how to set up an [nginx](http://nginx.org/en/docs/) reverse proxy for the SuzieQ GUI.
22+
23+
### Requirements
24+
25+
- nginx: [nginx installation guide](http://nginx.org/en/docs/install.html)
26+
- SuzieQ GUI running
27+
28+
### Configure and start nginx
29+
30+
Move into the nginx config directory
31+
32+
``` shell
33+
cd /etc/nginx
34+
```
35+
36+
!!! info
37+
Some of the commands below may need `sudo`
38+
39+
Fill the values into the following template and put it in a file under `/etc/nginx/sites-available/suzieq.conf`
40+
41+
```
42+
server {
43+
listen <PROXY_SERVER_PORT> default_server;
44+
listen [::]:<PROXY_SERVER_PORT> default_server;
45+
server_name <SERVER_NAME>;
46+
location /<PROXY_PATH> {
47+
proxy_pass <SUZIEQ_GUI_URL>;
48+
}
49+
# streamlit redirects config
50+
location /<PROXY_PATH>/streamlit-components-demo {
51+
proxy_pass <SUZIEQ_GUI_URL>/;
52+
}
53+
location ^~ /<PROXY_PATH>/static {
54+
proxy_pass <SUZIEQ_GUI_URL>/static/;
55+
}
56+
location ^~ /<PROXY_PATH>/healthz {
57+
proxy_pass <SUZIEQ_GUI_URL>/healthz;
58+
}
59+
location ^~ /<PROXY_PATH>/vendor {
60+
proxy_pass <SUZIEQ_GUI_URL>/vendor;
61+
}
62+
location ^~ /<PROXY_PATH>/component {
63+
proxy_pass <SUZIEQ_GUI_URL>/component;
64+
}
65+
66+
location /<PROXY_PATH>/stream {
67+
proxy_pass <SUZIEQ_GUI_URL>/stream;
68+
proxy_http_version 1.1;
69+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
70+
proxy_set_header Host $host;
71+
proxy_set_header Upgrade $http_upgrade;
72+
proxy_set_header Connection "upgrade";
73+
proxy_read_timeout 86400;
74+
}
75+
}
76+
```
77+
78+
The example below shows a configuration file shows how to create a reverse proxy
79+
to serve a SuzieQ GUI running on `http:localhost:8501` to be mapped to `http:localhost:80/suzieq`
80+
81+
```
82+
server {
83+
listen 80 default_server;
84+
listen [::]:80 default_server;
85+
server_name localhost;
86+
location /suzieq {
87+
proxy_pass http://127.0.0.1:8501;
88+
}
89+
# streamlit redirects config
90+
location /suzieq/streamlit-components-demo {
91+
proxy_pass http://127.0.0.1:8501/;
92+
}
93+
location ^~ /suzieq/static {
94+
proxy_pass http://127.0.0.1:8501/static/;
95+
}
96+
location ^~ /suzieq/healthz {
97+
proxy_pass http://127.0.0.1:8501/healthz;
98+
}
99+
location ^~ /suzieq/vendor {
100+
proxy_pass http://127.0.0.1:8501/vendor;
101+
}
102+
location ^~ /suzieq/component {
103+
proxy_pass http://127.0.0.1:8501/component;
104+
}
105+
106+
location /suzieq/stream {
107+
proxy_pass http://127.0.0.1:8501/stream;
108+
proxy_http_version 1.1;
109+
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
110+
proxy_set_header Host $host;
111+
proxy_set_header Upgrade $http_upgrade;
112+
proxy_set_header Connection "upgrade";
113+
proxy_read_timeout 86400;
114+
}
115+
}
116+
```
117+
118+
!!! warning
119+
These examples, do **NOT** provide any security.
120+
Please check the [nginx documentation](http://nginx.org/en/docs/) to know how add security to your nginx server
121+
122+
123+
Now copy the configuration from `/etc/nginx/sites-available` to `/etc/nginx/sites-enabled`.
124+
It is recommended to use a symbolic link.
125+
126+
``` shell
127+
ln -s /etc/nginx/sites-available/suzieq.conf /etc/nginx/sites-enabled/suzieq.conf
128+
```
129+
130+
Now restart the nginx service
131+
``` shell
132+
sudo systemctl restart nginx
133+
```
134+
135+
You can open you browser and connect to your configured nginx to see the SuzieQ GUI

docs/inventory.md

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ The new inventory is structured in 4 major pieces, explained in its own section:
1414
- `namespaces`: where you put together all the above. A namespace is be defined by a `source`, an `auth` and a `device`
1515

1616
Here is an example of an inventory file with a bunch of different options, but non-exhaustive, for each section:
17+
1718
```yaml
1819
sources:
1920
- name: netbox-instance-123
@@ -86,6 +87,7 @@ namespaces:
8687
- It is possible to [map different sources to the same namespace](#mapping-different-sources-to-the-same-namespace)
8788

8889
## <a name='sensitive-data'></a>Sensitive data
90+
8991
A sensitive data is an information that the user doesn't want to store in plain-text inside the inventory.
9092
For this reason, SuzieQ inventory now supports three different options to store these kind of informations
9193

@@ -94,6 +96,7 @@ For this reason, SuzieQ inventory now supports three different options to store
9496
- `ask`: the user can write the sensitive information on the stdin
9597

9698
Currently this method is used to specify passwords, passphrases and tokens.
99+
97100
## <a name='inventory-sources'></a>Sources
98101

99102
The device sources currently supported are:
@@ -121,9 +124,11 @@ Whenever a source has many fields in common with another, you don't have to rewr
121124
- suzieq-copy
122125
123126
```
127+
124128
### <a name='source-host-list'></a>Host list
125129

126130
The host list contains the IP address, the access method (SSH or REST), the IP address of the node, the user name, the type of OS if using REST and the access token such as a private key file. Here is an example of a native suzieq source type. For example (all possible combinations are shown for illustration):
131+
127132
```yaml
128133
- name: dc-01-native
129134
type: native # optional, if type is not present this is the default value
@@ -149,14 +154,16 @@ ansible-inventory --list > ansible.json
149154
```
150155

151156
Now you can set the path of the ansible inventory in the source:
157+
152158
```yaml
153159
- name: ansible-01
154160
type: ansible
155161
path: /path/to/ansible.json
156162
```
157163

164+
Since Ansible devices cannot really be split up, the device and auth sections apply to **all** the devices in the Ansible inventory file. This is a limitaion of the Ansible source input. We always assume ssh as the transport unless otherwise specified in the device section of the SuzieQ inventory file.
158165
!!! info
159-
The Ansible source assumes REST transport with Arista EOS and PanOs devices by default, and SSH for the others
166+
From 0.21.0, with Ansible inventories, the device type and transport are taken from the specification in the device section of the suzieq inventory file. You must specify the transport as rest if you want to use rest as the transport for EOS devices. By default, we assume ssh as the transport. For PANOS also, you must specify the device type and transport. Before version 0.21.0, Ansible inventory assumed REST as the transport for EOS, even if the user specified the transport as SSH in the device section.
160167

161168
### <a name='source-netbox'></a>Netbox
162169

@@ -173,6 +180,7 @@ Since Netbox is a _dynamic source_, the data are periodically pulled, the period
173180
If the user manually sets `ssl-verify: true` with an http netbox server, an error will be notified.
174181

175182
Here is an example of the configuration of a netbox type source:
183+
176184
```yaml
177185
- name: netbox-dc-01
178186
type: netbox
@@ -183,6 +191,7 @@ Here is an example of the configuration of a netbox type source:
183191
period: 3600 # How frequently Netbox should be polled
184192
ssl-verify: false # Netbox certificate validation will be skipped
185193
```
194+
186195
#### Selecting devices from Netbox
187196

188197
Starting from 0.19, it's possible to specify more than one tag to be matched, defining a list of one or more rules.
@@ -198,6 +207,7 @@ A device is polled by SuzieQ if it matches at least one of the defined rules.
198207
- alpha
199208
- bravo, charlie
200209
```
210+
201211
For example, the source above tells SuzieQ to select from Netbox all the devices having the `alpha` OR `bravo & charlie` tags.
202212

203213
!!!Warning
@@ -210,6 +220,7 @@ Netbox type source is capable to assign each device to a namespace which corresp
210220
To obtain this behaviour, we need to declare a `namespace` object with `name: netbox-sitename`.
211221

212222
Here is an example:
223+
213224
```yaml
214225
sources:
215226
- name: netbox-dc-01
@@ -270,6 +281,7 @@ In case you want to ignore the check of the device's key against the `known_host
270281
```
271282

272283
Moreover if all the devices inside a namespace run the same NOS, it is possible to specify it via the `devtype` option:
284+
273285
```yaml
274286
- name: eos-devices
275287
devtype: eos
@@ -294,13 +306,15 @@ This section is optional in case SuzieQ native and ansible source types. Here a
294306
Currently for both SSH and REST API, the only supported is username and password, therefore you will not be able to set api keys.
295307

296308
The simplest method is defining either username and password/private key.
309+
297310
```yaml
298311
- name: suzieq-user
299312
username: suzieq
300313
password: plain:pass
301314
```
302315

303316
In case a private key is used to authenticate:
317+
304318
```yaml
305319
- name: suzieq-user
306320
keyfile: path/to/private/key
@@ -326,6 +340,7 @@ A `cred-file` is an external file where you store credentials for all the device
326340
Each device credentials can be specified via its `hostname` or its `address`
327341
(with Netbox, it's encouraged the usage of `hostname`).
328342
The credential file should look like this:
343+
329344
```yaml
330345
- namespace: testing
331346
devices:
@@ -348,6 +363,7 @@ The credential file should look like this:
348363

349364
In the `namespaces` section sources, auths and devices can be put together to define namespaces.
350365
For example the following namespace will be defined by the source named `netbox-1`, the auths named `dc-01-credentials`, and the device named `ssh-jump-devs`:
366+
351367
```yaml
352368
namespaces:
353369
- name: example
@@ -430,15 +446,14 @@ Suppose we have this inventory valid for version 0.15.x:
430446
- url: ssh://[email protected]:2023 keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/server104/libvirt/private_key
431447
- url: https://[email protected] password=vagrant
432448
```
433-
The new inventory format consists of four sections (sources, auths, devices, namespaces) which are described above. We need to add the devices specified in the old inventory format in a new source inside the `sources` section and link it to a namespace.
434449

450+
The new inventory format consists of four sections (sources, auths, devices, namespaces) which are described above. We need to add the devices specified in the old inventory format in a new source inside the `sources` section and link it to a namespace.
435451

436452
Here is how the new format will look like:
437453

438454
!!! important
439455
Sections [auths](#auths) and [devices](#devices) are optional. See the full documentation to know how to use them.
440456

441-
442457
```yaml
443458
sources:
444459
- name: eos-source # namespace is defined below, this is only a name to be used as reference

docs/poller.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,13 @@ The inventory file format is covered in the [inventory page](./inventory.md).
2020

2121
To monitor the status of the poller, you can look at the log files created (by default) in the `/tmp` directory. All the aspects related to the creation/update of the inventory are logged into `sq-poller-controller.log`, while the each worker logs into `sq-poller-<x>.log` where `x` is the worker id.
2222

23+
!!! warning
24+
If you want to start the poller process as background task, remember to redirect the stdout to `/dev/null` otherwise the poller might crash when it tries to write something on the terminal.<br>
25+
`sq-poller -I inventory.yml >/dev/null &`
26+
2327
## Polling Modes
2428

25-
SuzieQ poller can run in either **continuous mode** or **snapshot mode**. In continuous mode, the poller runs forever, gathering data at the specified period from the supplied inventory file. Alternately, it can run in snapshot mode where it gathers the data just once and stops once it has gathered data from all the nodes in the inventory file. If we cannot gather data from a node, we do not persist in gathering data in the snapshot mode.
29+
SuzieQ poller can run in either **continuous mode** or **snapshot mode**. In continuous mode, the poller runs forever, gathering data at the specified period from the supplied inventory file. Alternately, it can run in snapshot mode where it gathers the data just once and stops once it has gathered data from all the nodes in the inventory file. If we cannot gather data from a node, we do not persist in gathering data in the snapshot mode.
2630

2731
The default mode is the continuous mode. To use the snapshot mode, add the option `--run-once=update`.
2832

@@ -92,7 +96,7 @@ have production versions of that code.
9296

9397
## Polling Period
9498

95-
When polling in continuous mode, SuzieQ uses the default period specified in the suzieq-cfg.yml [configuration file](./config_file.md) (you can change the default location via the `-c` option when launching the poller).
99+
When polling in continuous mode, SuzieQ uses the default period specified in the suzieq-cfg.yml [configuration file](./config_file.md) (you can change the default location via the `-c` option when launching the poller).
96100

97101
Independent of this, you can change the polling period of any individual service by modifying (or adding) the `period:<time in secs>` key-value pair to the service configuration file (located under lib/python\<python version\>/site-packages/suzieq/config) wherever SuzieQ is installed.
98102

@@ -128,7 +132,7 @@ jpietsch> sqpoller show status=fail namespace=dual-evpn
128132
354 dual-evpn server104 ospfNbr 1 [] [] [] [] [] 0 2020-06-17 05:14:40.980
129133
```
130134
131-
In this case the errors are because we aren't running any of those services (mlag, evpn etc.) on those nodes (server101, edge01 etc.).
135+
In this case the errors are because we aren't running any of those services (mlag, evpn etc.) on those nodes (server101, edge01 etc.).
132136
133137
## Database and Data Persistence
134138

docs/rate-limiting-AAA.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@ rest:
2020
# The certificates listed below are provided purely to get started, In any
2121
# secure deployment, these must be generated specifically for the site and
2222
# these lines uncommented and containing the location of the site-specific file.
23-
# rest_certfile: /suzieq/cert.pem
24-
# rest_keyfile: /suzieq/key.pem
23+
# rest-certfile: /suzieq/cert.pem
24+
# rest-keyfile: /suzieq/key.pem
2525
#
2626
API_KEY: 496157e6e869ef7f3d6ecb24a6f6d847b224ee4f
2727
logging-level: WARNING

0 commit comments

Comments
 (0)