README.md

## Intro
DCDB (DataCenter DataBase) is a database to collect various (sensor-)values of a datacenter for further analysis.
Harvesting of the data is task of the dcdbpusher.

# dcdbpusher

This is a general MQTT pusher which sends values of various sensors to the DCDB-database.
It ships with plugins for BACnet, IPMI, PDU(proprietary Power Delivery Unit, but could be used as XML plugin) ,percounter, SNMP and sysFS.

Build it by simply running
```bash
make
```
or alternatively use
```bash
make debug
```
within the `dcdbpusher` directory to build a version which will print additional debug-information during runtime.

The logic for the various sensors is encapsulated into plugins (shared dynamic libraries; the makefile will take care of compiling them for you). The dcdbpusher will dynamically open the libraries if they are specified in the [global configuration](#GC) file. Vice versa, if selected sensor-functionality, e.g. sysFS is not specified, the corresponding shared library libdcdbplugin_sysfs.so does not have to be present. 

You can run dcdbpusher by executing
```bash
./dcdbpusher path/to/configfile/
```
or run
```bash
./dcdbpusher -h
```
to print the help-section of dcdbpusher.

Dcdbpusher will check the given file-path for the global configuration file which has to be named `global.conf`.

### <a name="GC">Global Configuration</a>

The global configuration specifies various settings for dcdbpusher in general, e.g. which plugins should be loaded etc.
Please have a look at the provided `config/global.conf` example to get familiar with the file scheme. The example also forms a good starting point for writing a custom `global.conf`. The different sections and values are explained in the following table:

| Value | Explanation |
|:----- |:----------- |
| global | Wrapper structure for the global values.
| mqttBroker | Define address and port of the MQTT-broker which collects the messages (sensor values) send by dcdbpusher.
| mqttprefix | To not rewrite a full MQTT-topic for every sensor one can specify here a consistent prefix.
| threads | Specify how many threads should be created to handle the sensors async. Default value of threads is 1. Note that the MQTTPusher always starts an extra thread. So the actual number of started threads is always one more than defined here. Specifying not enough threads can result in a delay for some sensors until they are read.
| verbosity | Level of detail in the logfile (dcdb.log). Set to a value between 5 (all log-messages, default) and 0 (only fatal messages). NOTE: level of verbosity for the command-line log can be set via the -v flag independently when invoking dcdbpusher.
| daemonize | Set to 'true' if dcdbpusher should run detached as daemon. Default is false.
| tempdir | One can specify a writeable directory where dcdbpusher can write its temporary and logging files to. Default is the current (' ./ ' ) directory.
| cacheInterval | Define a time interval in seconds. The last sensor readings within this time interval will be kept. This value can be overwritten by plugins.
| | |
| restAPI | Bundles all values related to the RestAPI. See the corresponding [section](#restApi) for more information on supported functionality.
| address | Define address and port where the REST API server should run on.
| certificate | Provide the (path and) file which the HTTPS server should use as certificate.
| privateKey | Provide the (path and) file which should be used as corresponding private key for the certificate. If private key and certificate are stored in the same file one should nevertheless provide the path to the cert-file here again.
| dhFile | Provide the (path and) file where Diffie-Hellman parameters for the key exchange are stored.
| authkey | This struct is used to define authentication key tokens for the REST API. Within the struct, define which operations over the REST API are allowed for the token (e.g. PUTReq or GETReq). Each token must be unique.
| | |
| plugins | In this section one can specify the plugins which should be used.
| plugin _name_ | The plugin name is used to build the corresponding lib-name (e.g. sysfs --> libdcdbplugin_sysfs.1.0)
| path | Specify the path where the plugin (the shared library) is located. If left empty, dcdbpusher will look in the default lib-directories (usr/lib and friends) for the plugin-file.
| config | One can specify a separate config-file (including path to it) for the plugin to use. If not specified, dcdbpusher will look up pluginName.conf (e.g. sysfs.conf) in the same directory where global.conf is located.
| | |

Formats of the other sensor-specific config-files are explained in the corresponding [subsections](#IPMI). Example configuration-files can be found in the `config/` directory.


## <a name="restApi">REST API</a>

Dcdbpusher runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8000 on the localhost but the address can be changed in the [`global.conf`](#GC).

A HTTPS request to dcdbpusher should have the following format: `[GET|PUT] host:port[ressource]?[queries]`.
Tables with allowed ressources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. For every request at least the authentication token has to be appended as query. Multiple queries are separated by semicolons(';').

### Table of queries

| Key | Value | Explanation |
|:--- |:----- |:----------- |
| authkey | Your authentication token | The authentication token is required to verify that you are allowed to make this particular request. The authkey query must be included by every request
| interval | Time-value in [s] | Only for GET request of sensor readings average. One can (optionally) specify a custom time interval for which the average of sensor readings is calculated. By default, every sensor reading in the cache is used to calculate the average.


### Ressources for GET request

| Ressource | Explanation |
|:--------- |:----------- |
| /help | Responds with a small cheatsheet which presents all currently supported ressources.
| /plugins | (Discovery) Returns a list of all currently loaded plugins.
| /[plugin]/sensors | (Discovery) Returns a list of all sensors which belong to [plugin]. To find out which plugins are available one can request the `/plugins` ressource.
| /[plugin]/[sensor]/avg | Calculates and returns the average of the last sensor readings. Can be combined with the interval query.


### Ressources for PUT request

| Ressource | Explanation |
|:--------- |:----------- |
| /[plugin]/[action] | One can request to do a action on the specified plugin. Currently supported actions are `start`, `stop` and `reload` which start or stop the polling of the sensors of the plugin respectively triggers a reload of the plugin configuration. If a configuration reload is requested, all sensors of the plugin are stopped first. Then new sensors are created and started according to the configuration.


### Examples

Two examples for HTTPS requests:

```bash
GET https://localhost:8000/sysfs/freq1/avg?authkey=myToken;interval=15
```
```bash
PUT https://localhost:8000/bacnet/stop?authkey=myToken
```


## MQTT topic

For communication between the different DCDB-components (database, dcdbpusher) the [MQTT protocol](https://mqtt.org/) is used. In order to identify each sensor, everyone has to have a unique MQTT topic assigned. A MQTT topic for DCDB consists of exactly 128 bits (= 32 hex characters), not including '/' separators.

# Plugins

The core of dcdbpusher is responsible of collecting all the sensor values and sending them to the database. However, the main functionality of the sensors comes from the various plugins. Every plugin corresponds to a special sensor functionality.
Although every plugin requires different configuration parameters for its sensor functionality, some parameters are required equally for every sensor. The config-files for the plugins can follow two schemes: one where every sensor is defined on his own, and one (e.g. for IPMI and PDU) where sensors are aggregated to a higher level unit. The two possible schemes and uniformly needed sensor parameters shall be explained below. Configuration parameters which are only required specifically for sensors within a plugin are explained in the corresponding [subsections](#IPMI). For every plugin an example configuration file is supplied in the `/config` directory. One should use them as a starting point to write own configuration files. 
```
 Standalone sensors:							| Sensors divided into units:
------------------------------------------------+------------------------------------------------
												|
global {										|global {
	mqttprefix /00112233445566778899AABB0000	|	mqttprefix /00112233445566778899AABB0000
	cacheInterval 120							|	cacheInterval 120
}												|}
												|
SensorTemplate {								|SensorTemplate {
	sensor name {								|	sensor name {
		...										|		...
	}											|	}
												|
	...											|	...
}												|}
												|
sensors {										|Units {
 	sensor name1 {								|	sensorUnit u1 {
		default		A							|		unitParams		?
		interval	1000						|	
		mqttsuffix	0000						|		sensors {
		minValues	1							|			sensor name1 {
		params		?							|				default		A
	}											|				interval	1000
												|				mqttsuffix	000
	sensor name2 {								|				minValues	1
		...										|				params		?
	}											|			}
												|
	...											|			sensor name2 {
}												|				...
												|			}
												|			...
												|		}
												|	}
												|
												|	sensorUnit u2 {
												|		...
												|	}
												|
												|	...
												|}
```
Explanation of the values:

| Value | Explanation |
|:----- |:----------- |
| global | Here one can overwrite the global values defined in `global.conf`. The overwritten values are only used in the scope of the specific plugin. Overwriting of global values is completely optional. However, even if no global values are overwritten at least the `global{}` struct should be present.
| mqttprefix | Define separate MQTT prefix for this plugin.
| cacheInterval | Overwrite global caching interval with plugin specific value.
| | |
| SensorTemplate | Define template sensors to be used later in the configuration. This feature is mainly for convenience reasons. One is not obligated to define any template sensors. However it is required to at least define an empty SensorTemplate {} structure.
| sensor name | Every template sensor needs a name for future reference. A template sensor can define every value (including values specific to a plugin) a normal sensor can (see below).
| | |
| Units | Section to define different units where sensors may be aggregated to. Not every plugin makes use of this subdivision. Where it is used it the Units may be named different.
| sensorUnit u1 | If the unit subdivision is used, subunits need to be used and also need to be named.
| unitParams | One may be able to define special parameters specific to the subunits. They should be explained with the corresponding plugin if it makes use of subunits.
| | |
| sensors | Define in this section the sensors for the plugin/subunit
| sensor name | Every sensor needs a unique name
| default | Mention here the name of the template sensor to be used. The sensor will be initialized with the exact same values as the template sensor. This field can be left out if all required values for the sensor are defined manually.
| interval | Time in [ms] between two consecutive sensor reads. Default is 1000[ms] = 1[s]
| mqttsuffix | Suffix which is appended to the MQTT-prefix. Together with the prefix this should be a globally unique MQTT-topic for every sensor.
| minValues | Minimum number of sensor reads the sensor should gather before they are sent together to the database. Useful to reduce MQTT-overhead. Default is 1 (every sensor value is sent on his own).
| params | Every plugin requires additional configuration parameters. Those may be unique to every plugin and are explained in the corresponding subsections below.


## <a name="IPMI">IPMI</a>

The [IPMI](https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface) plugin enables dcdbpusher to collect sensor values offered by a baseboard management controller (BMC).

Explanation of the values specific for the IPMI plugin:

| Value | Explanation |
|:----- |:----------- |
| sessiontimeout | Session timeout value for the IPMI-connection
| retransmissiontimeout | Retransmission timeout value for the IPMI-connection
| username | For the remote IPMI-connection login credentials are required
| password | For the remote IPMI-connection login credentials are required
| mqttprefix | One can define a different MQTT-prefix per host
| cmd | One can define a raw IPMI-command (in hex-notation) to be sent. In this case also the start and stop fields for the response have to be defined. Alternatively, one can define the record-ID of the sensor (see below).
| start | Required if raw command is sent
| stop | Required if raw command is sent
| recordId | Define the record-ID of the sensor to be read. One can look up the corresponding record-IDs for every sensor with the "ipmi-sensors" command line tool (ships with the freeipmi-library). Alternatively, one can define a raw IPMI-command (see above).
| factor | One can specify a factor to scale the read value before it is stored in the database (to adjust precision).

## Perf-event

The Perfevent functionality is tasked with collecting data from the CPUs various performance counters (PMUs).
> NOTE &ensp;&ensp;&ensp; The perf-event plugin measures PMUs for all processes running on a specific CPU. Therefore a value of less than 1 is required in `/proc/sys/kernel/perf_event_paranoid`. Other values (>=1) restrict the access to PMUs. See this [footnote](#fn1) for additional information.

Explanation of the values specific for the perfevent plugin:

| Value | Explanation |
|:----- |:----------- |
| mqttsuffix | In the context of the perfevent plugin the mqttsuffixes are interpreted as start mqttsuffix. A perfsensor uses a separate MQTT-topic for every CPU. For every used CPU the mqttsuffix is incremented by one. However, dcdbpusher does not allow the same suffix twice! Example: For sensor A the mqttsuffix 00A8 is given and the system on which perfpusher is run has 8 cores. Then perfpusher uses the mqtt-suffixe 00A8, 00A9, 00AA, 00AB, 00AC, 00AD, 00AE, 00AF for the 8 cores. As they are already used by counter A, any other sensor must not use those. Counter B can e.g. start with mqttsuffix 00B0.
| type | Type of which the counter should be. Each type determines different possible values for the config-field. Possible type-values are described below.
| config | Together with the type-field config determines which performance counter should be read. Possible values and what they measure are listed below.
| cpus | One can define a comma-separated list of cpu numbers (also value ranges can be specified, e.g. 2-4 equals 2,3,4). The hardware counter will then be only opened on the specified cpus.


### type and config

(see the [perf_event_open man-page](http://man7.org/linux/man-pages/man2/perf_event_open.2.html) for more detailed explanations)

| Type | Config | Explanation |
|:----:|:------ |:----------- |
| PERF_TYPE_HARDWARE | | generalized hardware CPU events
| " | PERF_COUNT_HW_CPU_CYCLES | total cycles (affected by frequency scaling)
| " | PERF_COUNT_HW_INSTRUCTIONS | retired instructions
| " | PERF_COUNT_HW_CACHE_REFERENCES | cache accesses (usually last level)
| " | PERF_COUNT_HW_CACHE_MISSES | cache misses (usually last level)
| " | PERF_COUNT_HW_BRANCH_INSTRUCTIONS | retired branch instructions
| " | PERF_COUNT_HW_BRANCH_MISSES | mispredicted branch instructions
| " | PERF_COUNT_HW_BUS_CYCLES | bus cycles
| " | PERF_COUNT_HW_STALLED_CYCLES_FRONTEND | stalled cycles during issue
| " | PERF_COUNT_HW_STALLED_CYCLES_BACKEND  | stalled cycles during retirement
| " | PERF_COUNT_HW_REF_CPU_CYCLES | total cycles (unaffected by frequency scaling)
| | | |
| PERF_TYPE_SOFTWARE | | software events provided by the kernel
| " | PERF_COUNT_SW_CPU_CLOCK | reports CPU clock
| " | PERF_COUNT_SW_TASK_CLOCK | clock count specific to the running task
| " | PERF_COUNT_SW_PAGE_FAULTS | number of page faults
| " | PERF_COUNT_SW_CONTEXT_SWITCHES | count of context switches
| " | PERF_COUNT_SW_CPU_MIGRATIONS | times the process has migrated to a new CPU
| " | PERF_COUNT_SW_PAGE_FAULTS_MIN | number of minor page faults (no disk-I/O)
| " | PERF_COUNT_SW_PAGE_FAULTS_MAJ | number of major page faults (disk-I/O was required)
| " | PERF_COUNT_SW_ALIGNMENT_FAULTS | alignment faults when accessing unaligned memory
| " | PERF_COUNT_SW_EMULATION_FAULTS | number of unimplemented instructions which had to be emulated
| " | PERF_COUNT_SW_DUMMY | placeholder which counts nothing
| | | |
| PERF_TYPE_TRACEPOINT | | not yet implemented
| PERF_TYPE_HW_CACHE | | not yet implemented
| | | |
| PERF_TYPE_RAW | | user can define architecture-specific raw events here.
| " | *XXXX* | Config must be an hex value (without 0x prefix). See <sup>[2](#fn2)</sup>
| | | |
| PERF_TYPE_BREAKPOINT | --- | config not required, any values will be ignored. However config must still be specified (even if empty)

#### Footnotes

Taken from the [perf_event_open man-page](http://man7.org/linux/man-pages/man2/perf_event_open.2.html):

<a name="fn1">**1**</a>: &ensp; The pid and cpu arguments allow specifying which process and CPU to monitor:  
[...]  
pid == -1 and cpu >= 0  
This measures all processes/threads on the specified CPU. This requires CAP_SYS_ADMIN capability or a /proc/sys/kernel/perf_event_paranoid value of less than 1.

[...]

The perf_event_paranoid file can be set to restrict access to the performance counters.

| Value | Restriction |
|:-----:|:----------- |
| 2 | allow only user-space measurements (default since Linux 4.6) |
| 1 | allow both kernel and user measurements (default before Linux 4.6) |
| 0 | allow access to CPU-specific data but not raw trace-point samples |
| -1 | no restrictions |
	
The existence of the perf_event_paranoid file is the official method for determining if a kernel supports perf_event_open()

<a name="fn2">**2**</a>: &ensp; If type is *PERF_TYPE_RAW*, then a custom "raw" config value is needed. Most CPUs support events that are not covered by the "generalized" events. These are implementation defined; see your CPU manual (for example the Intel Volume 3B documentation or the AMD BIOS and Kernel Developer Guide). The libpfm4 library can be used to translate from the name in the architectural manual to the raw hex value perf_event_open() expects in this field.

## snmp

The SNMP plugin enables dcdbpusher to talk with devices which have an SNMP agent running and query requests from them. A SNMP sensor corresponds to a single value as identified by the unique OID. Sensors are aggregated by connections. See the exemplary snmp.conf file in the `config/` directory.
> NOTE &ensp;&ensp;&ensp; In the SNMP context the word privacy is used synonymously for encryption.

Explanation of the values specific for the SNMP plugin:

| Value | Explanation |
|:----- |:----------- |
| connection | An aggregating connection
| Type | Type of the SNMP application which runs on the device queried by the connection. Currently only the type Agent is supported.
| Host | Host name of the device which is to be queried.
| Port | The SNMP port should be usually 161. No changes should be required here.
| OIDPrefix | This OIDPrefix is used for all following sensors.
| |
| Version | Which SNMP version to use (either 2 (maps to 2c) or 3).
| Community | Which SNMP community to use (required only if version 2 is used).
| Username | Username to authenticate with (only required for version 3).
| SecLevel | The security level to be used (only required for version 3). Can be either `noAuthNoPriv` for no authentication and privacy ("privacy" is SNMPs synonym for encryption), `authNoPriv` for only authentication and `authPriv` for authentication and privacy.
| AuthProto | Which protocol to use for authentication (only required for version 3 and if SecLevel != noAuthNoPriv). Can be MD5 or SHA1.
| AuthKey | The passphrase for authentication (only required for version 3 and if SecLevel != noAuthNoPriv). Must be at least 8 characters long.
| PrivProto | Which protocol to use for privacy (only required for version 3 and if SecLevel = AuthPriv). Can be DES or AES.
| PrivKey | The passphrase for privacy encryption (only required for version 3 and if SecLevel = AuthPriv). Must be at least 8 characters long.
| mqttPart | Connection specific MQTT-part which is appended to the MQTT-prefix and succeded by the sensor specific suffix.
| |
| OID | OID suffix which together with the OIDPrefix forms the unique OID identifying a value to query.
| passphrase | has to be at least 8 characters long

## sysFS

SysFS sensors read data from sysFS files. The configuration file of the plugin corresponds to the generic plugin configuration with standalone sensors. Additionally for a sysFS sensor the following parameters are mandatory/possible:

Explanation of the values specific for the sysFS plugin:

| Value | Explanation |
|:----- |:----------- |
| path | Path to the sysFS file the sensor should read from. This parameter is mandatory.
| filter | One can define an optional filter if the sysFS file consists of more than only the sensor value. Please note the following points for filters: <br> 1.  The filter supports substitutions. For substitution sed syntax ("s/.../.../") is used. Therefore extended regular expressions (ERE) are used as regex-syntax. ERE is closest to Basic RE (BRE), which is actually used by sed, but requires less escaping. <br> 2.  If a \ ("backslash") is needed in the regex (for escaping), always use \\ ("double backslash") as the regex is read in as string and strings also escape with backslash <br> 3.  Whitespaces are actually used as value separators in the config files. If your filter requires whitespaces either use [[:space:]] in the regex or put it in quotation marks ("") <br> 4.  To be able to reference parts of the match (for substitution) use groups. Groups are created with parentheses. <br>  5.  If using character classes like [[:digit:]] always make sure to use double brackets ("[[" and "]]") or they will not be recognized. <br>  See [ERE-syntax](https://www.gnu.org/software/sed/manual/html_node/ERE-syntax.html#ERE-syntax) <br>  See [substitution syntax](http://www.boost.org/doc/libs/1_65_1/libs/regex/doc/html/boost_regex/format/sed_format.html)

## PDU

The Power Delivery Unit (PDU) plugin is in charge of sending a network-request to the PDUs and gathering specified sensor data from the XML-file response.

Explanation of the values specific for the PDU plugin:

| Value | Explanation |
|:----- |:----------- |
| host | Hostname and (optional) port where to fetch the XML-file with sensor data from. If no port is specified, 443 is used. The plugin requests the file via HTTPS.
| TTL | To avoid requesting a current XML-file every time a sensor wants to read his value, one can define a time to live (TTL) for the file here. A new XML-file is requested at the earliest if the TTL has expired. Default value is 1000[ms].
| request | Define the request to be sent to the host via HTTPS as a string. One should put the request in quotation marks (' " ') to enable the use of whitespaces within the request. Special characters (like usage of ' " ' within the request) should be escaped (' " ' --> ' \" '; ' \ ' --> ' \\\\ '; newline --> ' \n '; ...).
| path | Define a dot-separated path to the value to be read in the XML file. One can specify attribute values a node has to fulfil in brackets after the node. Even multiple (comma-separated) attributes can be given, however no whitespaces should be used (!) as they will not be filtered and could therefore be treat as part of the attributes name.

## BACnet

The BACnet plugin enables dcdbpusher to communicate and request data from devices which communicate via the BACnet protocol. A so called "read property" request is sent by the plugin to the BACnet devices as configured in the config file. The response value is then stored in the database. Usually one is only interested in collecting the current reading of a BACnet device (property PROP_PRESENT_VALUE, ID 85). However, also reading of other properties is supported.
> NOTE &ensp;&ensp;&ensp; On startup BACnet plugin does no device discovery. Instead it relies on the user providing a file with addresses of all required BACnet devices. One can generate such an address-file for example by using the `bacwi` demo tool provided by the BACnet-Stack.

Explanation of the values specific for the BACnet plugin:

| Value | Explanation |
|:----- |:----------- |
| address_cache | (Path to and) filename of the address cache file where the addresses of BACnet devices are stored (as noted above).
| interface | Network interface (IPv4) which is to be used by the plugin to send its "Read Property" requests.
| port | Port to use on the interface
| timeout |	Value of µ-seconds to wait for a response packet.
| apdu_timeout | Value of µ-seconds before sending a request times out.
| apdu_retries | How often should sending a request be retried.
| templates | One can define template properties in this section for convenience.
| factor | Described in the section for the [IPMI-plugin](#IPMI).
| devices | Starts the part in the config file where the actual BACnet devices are configured. A BACnet device consists of multiple nested parts: device > objects > properties.
| mqttPart | One can define a partial MQTT topic on each nesting level, which will be appended to the MQTT prefix. Together with the mqttsuffix of a property this will make up the whole MQTT topic of a property (aka sensor). MQTT topic = mqttPrefix + mqttPart (device) + mqttPart (object) + mqttsuffix.
| instance (device) | Instance of the BACnet-device.
| type | Type of the object within the device.
| instance (object) | Instance of the object within the device.
| id | ID of the property to be read from the BACnet device-object. Assignment of numbers to properties is done according to the enum as defined in `bacenum.h`.

## Opa (Intel Omni-Path Archtiecture)

The Opa plugin enables dcdbpusher to query various counters from omni-path interconnects.

Explanation of the values specific for the Opa plugin:

| Value | Explanation |
|:----- |:----------- |
| hfiNum | Number of which omni-path Host Fabric Interface to query (starting with 1)
| portNum | Number of which omni-path port to query (starting with 1)
| cntData | Name which data counter to query. A list of possible values can be found below.

> NOTE &ensp;&ensp;&ensp; Specifying `hfiNum` or `portNum` within a template sensor is currently not supported.

### counterData

Possible values for cntData:
* portXmitData
* portRcvData
* portXmitPkts
* portRcvPkts
* portMulticastXmitPkts
* portMulticastRcvPkts
* localLinkIntegrityErrors
* fmConfigErrors
* portRcvErrors
* excessiveBufferOverruns
* portRcvConstraintErrors
* portRcvSwitchRelayErrors
* portXmitDiscards
* portXmitConstraintErrors
* portRcvRemotePhysicalErrors
* swPortCongestion
* portXmitWait
* portRcvFECN
* portRcvBECN
* portXmitTimeCong
* portXmitWastedBW
* portXmitWaitData
* portRcvBubble
* portMarkFECN
* linkErrorRecovery
* linkDowned
* uncorrectableErrors

## Writing own plugins
Try out the `pluginGenerator/generatePlugin.sh` script!
TODO!

#### TODOS
* explain sensor group concept
* rework pdu plugin: should be more generic XML plugin
* more about mqtt-topic