Commit 9e84de5d authored by Micha Mueller's avatar Micha Mueller
Browse files

Add table of contents to README

parent 3938c62a
## Intro
# Table of contents
1. [Introduction](#introduction)
2. [dcdbpusher](#dcdbpusher)
1. [Global Configuration](#globalConfiguration)
2. [Rest API](#restApi)
1. [Table of queries](#tableOfQueries)
2. [Ressources for GET request](#ressourcesGET)
3. [Ressources for PUT request](#ressourcesPUT)
4. [Examples](#restExamples)
3. [MQTT topic](#mqttTopic)
3. [Plugins](#plugins)
1. [IPMI](#ipmi)
2. [Perf-event](#perf)
1. [type and config](#perfTypeConfig)
2. [Footnotes](#perfFootnotes)
3. [SNMP](#snmp)
4. [SysFS](#sysfs)
5. [PDU](#pdu)
6. [BACnet](#bacnet)
7. [OPA](#opa)
1. [counterData](#opaCounterData)
8. [Writing own plugins](#writingOwnPlugins)
3. [TODOS](#todos)
## Introduction <a name="introduction"></a>
DCDB (DataCenter DataBase) is a database to collect various (sensor-)values of a datacenter for further analysis.
Harvesting of the data is task of the dcdbpusher.
# dcdbpusher
# dcdbpusher <a name="dcdbpusher"></a>
This is a general MQTT pusher which sends values of various sensors to the DCDB-database.
It ships with plugins for BACnet, IPMI, PDU(proprietary Power Delivery Unit, but could be used as XML plugin) ,percounter, SNMP and sysFS.
......@@ -31,7 +55,7 @@ to print the help-section of dcdbpusher.
Dcdbpusher will check the given file-path for the global configuration file which has to be named `global.conf`.
### <a name="GC">Global Configuration</a>
### Global Configuration <a name="globalConfiguration"></a>
The global configuration specifies various settings for dcdbpusher in general, e.g. which plugins should be loaded etc.
Please have a look at the provided `config/global.conf` example to get familiar with the file scheme. The example also forms a good starting point for writing a custom `global.conf`. The different sections and values are explained in the following table:
......@@ -60,17 +84,17 @@ Please have a look at the provided `config/global.conf` example to get familiar
| config | One can specify a separate config-file (including path to it) for the plugin to use. If not specified, dcdbpusher will look up pluginName.conf (e.g. sysfs.conf) in the same directory where global.conf is located.
| | |
Formats of the other sensor-specific config-files are explained in the corresponding [subsections](#IPMI). Example configuration-files can be found in the `config/` directory.
Formats of the other sensor-specific config-files are explained in the corresponding [subsections](#ipmi). Example configuration-files can be found in the `config/` directory.
## <a name="restApi">REST API</a>
## REST API <a name="restApi"></a>
Dcdbpusher runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8000 on the localhost but the address can be changed in the [`global.conf`](#GC).
Dcdbpusher runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8000 on the localhost but the address can be changed in the [`global.conf`](#globalConfiguration).
A HTTPS request to dcdbpusher should have the following format: `[GET|PUT] host:port[ressource]?[queries]`.
Tables with allowed ressources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. For every request at least the authentication token has to be appended as query. Multiple queries are separated by semicolons(';').
### Table of queries
### Table of queries <a name="tableOfQueries"></a>
| Key | Value | Explanation |
|:--- |:----- |:----------- |
......@@ -78,7 +102,7 @@ Tables with allowed ressources sorted by REST methods can be found below. A quer
| interval | Time-value in [s] | Only for GET request of sensor readings average. One can (optionally) specify a custom time interval for which the average of sensor readings is calculated. By default, every sensor reading in the cache is used to calculate the average.
### Ressources for GET request
### Ressources for GET request <a name="ressourcesGET"></a>
| Ressource | Explanation |
|:--------- |:----------- |
......@@ -88,14 +112,14 @@ Tables with allowed ressources sorted by REST methods can be found below. A quer
| /[plugin]/[sensor]/avg | Calculates and returns the average of the last sensor readings. Can be combined with the interval query.
### Ressources for PUT request
### Ressources for PUT request <a name="ressourcesPUT"></a>
| Ressource | Explanation |
|:--------- |:----------- |
| /[plugin]/[action] | One can request to do a action on the specified plugin. Currently supported actions are `start`, `stop` and `reload` which start or stop the polling of the sensors of the plugin respectively triggers a reload of the plugin configuration. If a configuration reload is requested, all sensors of the plugin are stopped first. Then new sensors are created and started according to the configuration.
### Examples
### Examples <a name="restExamples"></a>
Two examples for HTTPS requests:
......@@ -106,7 +130,7 @@ GET https://localhost:8000/sysfs/freq1/avg?authkey=myToken;interval=15
PUT https://localhost:8000/bacnet/stop?authkey=myToken
```
## <a name="MQTT">MQTT topic</a>
## MQTT topic <a name="mqttTopic></a>
For communication between the different DCDB-components (database, dcdbpusher) the [MQTT protocol](https://mqtt.org/) is used. In order to identify each sensor, everyone has to have a unique MQTT topic assigned. A MQTT topic for DCDB consists of exactly 128 bits (= 32 hex characters), not including '/' separators. The topic for a sensor is build by appending up to 4 parts:
1. mqttprefix (e.g. /00112233445566778899AA)
......@@ -116,10 +140,10 @@ For communication between the different DCDB-components (database, dcdbpusher) t
Then the topic for the sensor is /00112233445566778899AA/BB/1122/3344.
# <a name ="Plugins">Plugins</a>
# Plugins <a name ="plugins"></a>
The core of dcdbpusher is responsible of collecting all the values read by the sensors and sending them to the database. However, the main functionality of the sensors comes from the various plugins. Every plugin corresponds to a special sensor functionality.
All the different plugins share some same general principles in common regarding the sensor structure and configuration. Those principles should also be obeyed when [writing own plugins](#WOP).
All the different plugins share some same general principles in common regarding the sensor structure and configuration. Those principles should also be obeyed when [writing own plugins](#writingOwnPlugins).
1. There are three hierarchical levels (from bottom up):
1. Sensors
2. Groups
......@@ -133,10 +157,10 @@ All the different plugins share some same general principles in common regarding
2. Groups hold all attributes which multiple sensors belonging to it share in common. Common group attributes:
* __interval__ (Time in [ms] between two consecutive sensor reads. Default is 1000[ms] = 1[s])
* __minValues__ (Minimum number of sensor reads the sensors in a group should gather before they are sent together to the database. Useful to reduce MQTT-overhead. Default is 1 (every sensor value is sent on its own))
* __mqttPart__ (Part for the [mqtt-topic](#MQTT) all sensors in this group should share in common)
* __mqttPart__ (Part for the [mqtt-topic](#mqttTopic) all sensors in this group should share in common)
* __default__ (One can define the name of a template group (see below) whose values and sensors should be used as default)
3. Sensors hold only those attributes which are necessary to uniquely identify the target sensor. Common base attributes:
* __mqttsuffix__ (to make its [mqtt-topic](#MQTT) unique)
* __mqttsuffix__ (to make its [mqtt-topic](#mqttTopic) unique)
5. Be aware that naming of sensor/group/entity is not fixed. A plugin developer can name them as he likes, e.g. counter/multicounter/host.
6. It is possible to define template groups or entities in the config file, but not template sensors (as a sensor should only consists of attributs which make him unique this would not be too useful). To specify a template group/entity simply prefix its definition with `template_` (see the example below). You can reference them later by using the `default` attribute. A template entity can consist of groups and these in turn can consist of sensors. When using a template, all of its attribute values are copied to the actual sensor. Copied attributes can be overwritten in the actual entity/sensor (some of them even should be overwritten, e.g. the mqttPart). However, groups/sensors associated with a template are copied to the actual entity/group and can NOT be overwritten. One can specify further groups/sensors which are then added to those copied from the template. Template entitys/groups itself or sensors within them are never used in live operation of the plugin. They are purely cosmetic for convenient configuration.
......@@ -230,7 +254,7 @@ entity ent1 {
```
One should have noticed the global section in the examples which was not mentioned before. In this section the user can (but is not obligated to) overwrite values from the `global.conf` for this plugin or specify other settings which are global for this plugin.
## <a name="IPMI">IPMI</a>
## IPMI <a name="ipmi"></a>
The [IPMI](https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface) plugin enables dcdbpusher to collect sensor values offered by a baseboard management controller (BMC).
......@@ -248,7 +272,7 @@ Explanation of the values specific for the IPMI plugin:
| recordId | Define the record-ID of the sensor to be read. One can look up the corresponding record-IDs for every sensor with the "ipmi-sensors" command line tool (ships with the freeipmi-library). Alternatively, one can define a raw IPMI-command (see above).
| factor | One can specify a factor to scale the read value before it is stored in the database (to adjust precision).
## Perf-event
## Perf-event <a name="perf"></a>
The Perfevent functionality is tasked with collecting data from the CPUs various performance counters (PMUs).
> NOTE &ensp;&ensp;&ensp; The perf-event plugin measures PMUs for all processes running on a specific CPU. Therefore a value of less than 1 is required in `/proc/sys/kernel/perf_event_paranoid`. Other values (>=1) restrict the access to PMUs. See this [footnote](#fn1) for additional information.
......@@ -263,7 +287,7 @@ Explanation of the values specific for the perfevent plugin:
| cpus | One can define a comma-separated list of cpu numbers (also value ranges can be specified, e.g. 2-4 equals 2,3,4). The hardware counter will then be only opened on the specified cpus.
### type and config
### type and config <a name="perfTypeConfig"></a>
(see the [perf_event_open man-page](http://man7.org/linux/man-pages/man2/perf_event_open.2.html) for more detailed explanations)
......@@ -301,7 +325,7 @@ Explanation of the values specific for the perfevent plugin:
| | | |
| PERF_TYPE_BREAKPOINT | --- | config not required, any values will be ignored. However config must still be specified (even if empty)
#### Footnotes
#### Footnotes <a name="perfFootnotes"></a>
Taken from the [perf_event_open man-page](http://man7.org/linux/man-pages/man2/perf_event_open.2.html):
......@@ -325,7 +349,7 @@ The existence of the perf_event_paranoid file is the official method for determi
<a name="fn2">**2**</a>: &ensp; If type is *PERF_TYPE_RAW*, then a custom "raw" config value is needed. Most CPUs support events that are not covered by the "generalized" events. These are implementation defined; see your CPU manual (for example the Intel Volume 3B documentation or the AMD BIOS and Kernel Developer Guide). The libpfm4 library can be used to translate from the name in the architectural manual to the raw hex value perf_event_open() expects in this field.
## snmp
## snmp <a name="snmp"></a>
The SNMP plugin enables dcdbpusher to talk with devices which have an SNMP agent running and query requests from them. A SNMP sensor corresponds to a single value as identified by the unique OID. Sensors are aggregated by connections. See the exemplary snmp.conf file in the `config/` directory.
> NOTE &ensp;&ensp;&ensp; In the SNMP context the word privacy is used synonymously for encryption.
......@@ -353,7 +377,7 @@ Explanation of the values specific for the SNMP plugin:
| OID | OID suffix which together with the OIDPrefix forms the unique OID identifying a value to query.
| passphrase | has to be at least 8 characters long
## sysFS
## sysFS <a name="sysfs"></a>
SysFS sensors read data from sysFS files. The configuration file of the plugin corresponds to the generic plugin configuration with standalone sensors. Additionally for a sysFS sensor the following parameters are mandatory/possible:
......@@ -364,7 +388,7 @@ Explanation of the values specific for the sysFS plugin:
| path | Path to the sysFS file the sensor should read from. This parameter is mandatory.
| filter | One can define an optional filter if the sysFS file consists of more than only the sensor value. Please note the following points for filters: <br> 1. The filter supports substitutions. For substitution sed syntax ("s/.../.../") is used. Therefore extended regular expressions (ERE) are used as regex-syntax. ERE is closest to Basic RE (BRE), which is actually used by sed, but requires less escaping. <br> 2. If a \ ("backslash") is needed in the regex (for escaping), always use \\ ("double backslash") as the regex is read in as string and strings also escape with backslash <br> 3. Whitespaces are actually used as value separators in the config files. If your filter requires whitespaces either use [[:space:]] in the regex or put it in quotation marks ("") <br> 4. To be able to reference parts of the match (for substitution) use groups. Groups are created with parentheses. <br> 5. If using character classes like [[:digit:]] always make sure to use double brackets ("[[" and "]]") or they will not be recognized. <br> See [ERE-syntax](https://www.gnu.org/software/sed/manual/html_node/ERE-syntax.html#ERE-syntax) <br> See [substitution syntax](http://www.boost.org/doc/libs/1_65_1/libs/regex/doc/html/boost_regex/format/sed_format.html)
## PDU
## PDU <a name="pdu"></a>
The Power Delivery Unit (PDU) plugin is in charge of sending a network-request to the PDUs and gathering specified sensor data from the XML-file response.
......@@ -377,7 +401,7 @@ Explanation of the values specific for the PDU plugin:
| request | Define the request to be sent to the host via HTTPS as a string. One should put the request in quotation marks (' " ') to enable the use of whitespaces within the request. Special characters (like usage of ' " ' within the request) should be escaped (' " ' --> ' \" '; ' \ ' --> ' \\\\ '; newline --> ' \n '; ...).
| path | Define a dot-separated path to the value to be read in the XML file. One can specify attribute values a node has to fulfil in brackets after the node. Even multiple (comma-separated) attributes can be given, however no whitespaces should be used (!) as they will not be filtered and could therefore be treat as part of the attributes name.
## BACnet
## BACnet <a name="bacnet"></a>
The BACnet plugin enables dcdbpusher to communicate and request data from devices which communicate via the BACnet protocol. A so called "read property" request is sent by the plugin to the BACnet devices as configured in the config file. The response value is then stored in the database. Usually one is only interested in collecting the current reading of a BACnet device (property PROP_PRESENT_VALUE, ID 85). However, also reading of other properties is supported.
> NOTE &ensp;&ensp;&ensp; On startup BACnet plugin does no device discovery. Instead it relies on the user providing a file with addresses of all required BACnet devices. One can generate such an address-file for example by using the `bacwi` demo tool provided by the BACnet-Stack.
......@@ -393,14 +417,14 @@ Explanation of the values specific for the BACnet plugin:
| apdu_timeout | Value of µ-seconds before sending a request times out.
| apdu_retries | How often should sending a request be retried.
| templates | One can define template properties in this section for convenience.
| factor | Described in the section for the [IPMI-plugin](#IPMI).
| factor | Described in the section for the [IPMI-plugin](#ipmi).
| devices | Starts the part in the config file where the actual BACnet devices are configured. A BACnet device consists of multiple nested parts: device > objects > properties.
| instance (device) | Instance of the BACnet-device.
| type | Type of the object within the device.
| instance (object) | Instance of the object within the device.
| id | ID of the property to be read from the BACnet device-object. Assignment of numbers to properties is done according to the enum as defined in `bacenum.h`.
## Opa (Intel Omni-Path Architecture)
## Opa (Intel Omni-Path Architecture) <a name="opa"></a>
The Opa plugin enables dcdbpusher to query various counters from omni-path interconnects.
......@@ -412,7 +436,7 @@ Explanation of the values specific for the Opa plugin:
| portNum | Number of which omni-path port to query (starting with 1)
| cntData | Name which data counter to query. A list of possible values can be found below.
### counterData
### counterData <a name="opaCounterData"></a>
Possible values for cntData:
* portXmitData
......@@ -443,11 +467,11 @@ Possible values for cntData:
* linkDowned
* uncorrectableErrors
## <a name="WOP">Writing own plugins</a>
First make sure you read the [plugins](#Plugins) section.
## Writing own plugins <a name="writingOwnPlugins"></a>
First make sure you read the [plugins](#plugins) section.
Try out the `pluginGenerator/generatePlugin.sh` script!
TODO!
#### TODOS
#### TODOS <a name="todos"></a>
* explain special cases in the code for every plugin?
* more about mqtt-topic
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment