Commit 4d191371 authored by Alessio Netti's avatar Alessio Netti

Updates to documentation

parent 5a544c2d
......@@ -6,11 +6,11 @@ The *Data Center* *Data Base* (DCDB) is a modular, continuous and holistic monit
* _Collect_ _Agent_: The collect agent functions as intermediary between one storage backend and one or multiple pushers.
* _Pusher_: The main part for collecting data is the pusher framework. It allows to run arbitrary data collection plugins and pushes the data to the collect agent via MQTT messages.
Other features included by DCDB:
* libdcdb: a library to access the storage backend with external tools
* a set of tools leveraging libdcdb (e.g. a command line tool to query the storage backend)
* built in data analytics framework
* pusher plugins for a variety of typical HPC data sources
Other features included in DCDB:
* libdcdb: a library to access the storage backend with external tools.
* A set of tools leveraging libdcdb (e.g. a command line tool to query the storage backend).
* Built-in data analytics framework.
* Pusher plugins for a variety of typical HPC data sources.
## Build, install and run
......@@ -33,7 +33,7 @@ $ make depsinstall install
$ make depsinstall install doc
```
To run DCDB locally with the included default config files:
To run DCDB locally with the included default configuration files:
```bash
//Change to the DCDB directory you created during installation
......@@ -60,7 +60,7 @@ DCDB was created at Leibniz Supercomputing Centre (LRZ).
For questions and/or suggestions please contact info@dcdb.it
Copyright (C) 2011-2019 Leibniz Supercomputing Centre
Copyright (C) 2011-2020 Leibniz Supercomputing Centre
DCDB (except for the libdcdb parts) is licensed under the terms
and conditions of the GNU GENERAL PUBLIC LICENSE (Version 2 or later).
......
......@@ -16,7 +16,7 @@
5. [MQTT Topics](#mqttTopics)
6. [Pipelining Operators](#pipelining)
3. [Rest API](#restApi)
1. [List of ressources](#listOfRessources)
1. [List of resources](#listOfResources)
2. [Examples](#restExamples)
3. [Plugins](#plugins)
1. [Aggregator Plugin](#averagePlugin)
......@@ -29,7 +29,7 @@
7. [Tester Plugin](#testerPlugin)
4. [Sink Plugins](#sinkplugins)
1. [File Sink Plugin](#filesinkPlugin)
2. [Writing Plugins](#writingPlugins)
5. [Writing Plugins](#writingPlugins)
### Additional Resources
......@@ -528,13 +528,13 @@ Wintermute provides a REST API that can be used to perform various management op
API is functionally identical to that of DCDB Pusher, and is hosted at the same address. All requests that are targeted
at the data analytics framework must have a resource path starting with _/analytics_.
### List of ressources <a name="listOfRessources"></a>
### List of resources <a name="listOfResources"></a>
Prefix `/analytics` left out!
<table>
<tr>
<td colspan="2"><b>Ressource</b></td>
<td colspan="2"><b>Resource</b></td>
<td colspan="2">Description</td>
</tr>
<tr>
......@@ -962,7 +962,7 @@ Additionally, input sensors in sinks accept the following parameters:
|:----- |:----------- |
| path | The path to which the sensors's readings should be written. It is interpreted as described above for the _autoName_ attribute.
## Writing DCDB Analytics Plugins <a name="writingPlugins"></a>
# Writing Wintermute Plugins <a name="writingPlugins"></a>
Generating a DCDB Wintermute plugin requires implementing a _Operator_ and _Configurator_ class which contain all logic
tied to the specific plugin. Such classes should be derived from _OperatorTemplate_ and _OperatorConfiguratorTemplate_
respectively, which contain all plugin-agnostic configuration and runtime features. Please refer to the documentation
......
# Collect Agent
### Table of contents
1. [Introduction](#introduction)
2. [collectagent](#collectagent)
1. [Configuration](#configuration)
2. [Rest API](#restApi)
1. [List of resources](#listOfResources)
2. [Examples](#restExamples)
## Introduction <a name="introduction"></a>
DCDB (Data Center Data Base) is a database to collect sensor data associated to data center operation for further analysis.
The collectagent serves the role of data broker in the DCDB hierarchy.
# collectagent <a name="collectagent"></a>
The collectagent acts as an MQTT broker that subscribes to all topics, and has the task of forwarding the data sent by DCDB Pushers into an Apache Cassandra database.
It will try to batch database inserts, depending on the incoming MQTT packets, in order to optimize performance.
Build it by simply running
```bash
make
```
or alternatively use
```bash
make debug
```
within the `collectagent` directory to build a version which will print additional debug-information during runtime.
You can run a collectagent by executing
```bash
./collectagent path/to/configfile/
```
or run
```bash
./collectagent -h
```
to print the help-section of the collectagent.
The collectagent will check the given file-path for the global configuration file which has to be named `collectagent.conf`.
### Configuration <a name="configuration"></a>
The configuration specifies various general settings for the collectagent. Please have a look at the provided `config/collectagent.conf` example to get familiar with the file scheme. The different sections and values are explained in the following table:
| Value | Explanation |
|:----- |:----------- |
| global | Wrapper structure for the global values.
| threads | Specifies how many threads should be created for the local thread pool. This is used by the Wintermute plugins as well as the REST API. Default value of threads is 1.
| messageThreads | Number of threads to be used by the MQTT broker.
| messageSlots | Maximum number of connections allowed per MQTT thread.
| cacheInterval | Size (in seconds) of the local cache for each incoming sensor.
| verbosity | Level of detail in the log file. Set to a value between 5 (all log-messages, default) and 0 (only fatal messages). NOTE: level of verbosity for the command-line log can be set via the -v flag independently when invoking the collectagent.
| daemonize | Set to 'true' if the collectagent should run detached as daemon. Default is false.
| tempdir | One can specify a writeable directory where the collectagent can write its temporary and logging files to. Default is the current (' ./ ' ) directory.
| statistics | Boolean. If true, the collectagent will periodically print out statistic about data and insert rates.
| mqttListenAddresss | Address on which the MQTT broker must listen for incoming connections. Default is 127.0.0.1:1883.
| cleaningInterval | Periodic interval (in seconds) over which stale entries are purged from the sensor cache.
| | |
| cassandra | Wrapper for the Cassandra DB settings.
| address | Define address and port of the Cassandra DB server into which sensor data should be stored. Default is 127.0.0.1:9042.
| user | Username for Cassandra authentication. Default is empty.
| password | Password for Cassandra authentication. Default is empty.
| numThreadsIo | Number of I/O threads in the Cassandra driver. Default is 1.
| queueSizeIo | Size of the request queue for each I/O thread. Default is 4096.
| coreConnPerHost | Maximum number of connection per Cassandra database. Default is 1.
| ttl | Default *time to live* to be used for sensor data, if none is provided in the respective metadata.
| debugLog | Outputs verbose error messages if enabled. Default is false.
| | |
| restAPI | Bundles all values related to the RestAPI. See the corresponding [section](#restApi) for more information on supported functionality.
| address | Define (IP-)address and port where the REST API server should run on. Default is 127.0.0.1:8080.
| certificate | Provide the (path and) file which the HTTPS server should use as certificate.
| privateKey | Provide the (path and) file which should be used as corresponding private key for the certificate. If private key and certificate are stored in the same file one should nevertheless provide the path to the cert-file here again.
| dhFile | Provide the (path and) file where Diffie-Hellman parameters for the key exchange are stored.
| user | This struct is used to define username and password for users of the REST API, along with their respective allowed REST operations (i.e., GET, PUT, POST).
| | |
An example configuration file can be found in the `config/` directory. An explanation of how to deploy Wintermute
data analytics plugins can be found in the corresponding readme document.
## REST API <a name="restApi"></a>
The collectagent runs an HTTPS server which provides some functionality to be controlled over a RESTful API.
The API is by default hosted at port 8080 on 127.0.0.1 but the address can be changed in [`collectagent.conf`](#configuration).
An HTTPS request to the collectagent should have the following format: `[GET|PUT|POST] host:port[resource]?[queries]`.
Tables with allowed resources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons (';'). For all requests (except /help) basic authentication credentials must be provided.
### List of resources <a name="listOfResources"></a>
<table>
<tr>
<td colspan="2"><b>Resource</b></td>
<td colspan="2">Description</td>
</tr>
<tr>
<td>Query</td>
<td>Value</td>
<td>Opt.</td>
<td>Description</td>
</tr>
</table>
<table>
<tr>
<td colspan="2"><b>GET /help</b></td>
<td colspan="2">Return a cheatsheet of possible REST API endpoints.</td>
</tr>
<tr>
<td colspan="4">No queries.</td>
</tr>
</table>
<table>
<tr>
<td colspan="2"><b>GET /average</b></td>
<td colspan="2">Get the average of the last readings of a sensor in the local cache. Also allows access to analytics sensors.</td>
</tr>
<tr>
<td>sensor</td>
<td>All sensor names that are available.</td>
<td>No</td>
<td>Specify a sensor stored in the local cache.</td>
</tr>
<tr>
<td>interval</td>
<td>Number of seconds.</td>
<td>Yes</td>
<td>Use only readings more recent than (now - interval) for average calculation. Defaults to zero, i.e. all cached sensor readings are included in average calculation.</td>
</tr>
</table>
<table>
<tr>
<td colspan="2"><b>PUT /quit</b></td>
<td colspan="2">Exits the collectagent with a user-specified return code.</td>
</tr>
<tr>
<td>code</td>
<td>Return code.</td>
<td>Yes</td>
<td>Return code to be used when exiting.</td>
</tr>
</table>
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; Opt. = Optional
### Examples <a name="restExamples"></a>
Two examples for HTTPS requests (authentication credentials not shown):
```bash
GET https://localhost:8080/average?sensor=/clust06/node55/freq1;interval=15
```
```bash
PUT https://localhost:8080/quit?code=11
```
\ No newline at end of file
......@@ -5,7 +5,7 @@
2. [dcdbpusher](#dcdbpusher)
1. [Global Configuration](#globalConfiguration)
2. [Rest API](#restApi)
1. [List of ressources](#listOfRessources)
1. [List of resources](#listOfResources)
2. [Examples](#restExamples)
3. [MQTT topic](#mqttTopic)
3. [Plugins](#plugins)
......@@ -31,7 +31,7 @@ Harvesting of the data is task of the dcdbpusher.
# dcdbpusher <a name="dcdbpusher"></a>
This is a general MQTT pusher which sends values of various sensors to the DCDB-database.
It ships with plugins for BACnet, IPMI, PDU(proprietary Power Delivery Unit, but could be used as XML plugin), perfcounter, SNMP and sysFS.
It ships with plugins for BACnet, IPMI, PDU (proprietary Power Delivery Unit, but could be used as XML plugin), perfcounter, SNMP and sysFS, among others.
Build it by simply running
```bash
......@@ -68,8 +68,8 @@ Please have a look at the provided `config/dcdbpusher.conf` example to get famil
| mqttBroker | Define address and port of the MQTT-broker which collects the messages (sensor values) send by dcdbpusher.
| mqttprefix | To not rewrite a full MQTT-topic for every sensor one can specify here a consistent prefix.
| sensorpattern | pattern used to perform automatic sensor name publishing. See the corresponding [section](#autopublish) for more information.
| threads | Specify how many threads should be created to handle the sensors async. Default value of threads is 1. Note that the MQTTPusher always starts an extra thread. So the actual number of started threads is always one more than defined here. Specifying not enough threads can result in a delay for some sensors until they are read.
| maxMsgNum | To avoid publishing too many MQTT messages at once you can define here a maximum count of values that are published in one turn. After reaching this limit the MQTTPusher will be forced to sleep for a short time before continuing.
| threads | Specify how many threads should be created to handle the sensors asynchronously, as well as the Wintermute plugins. Default value of threads is 1. Note that the MQTT Pusher always starts an extra thread. So the actual number of started threads is always one more than defined here. Specifying not enough threads can result in a delay for some sensors until they are read.
| maxMsgNum | To avoid publishing too many MQTT messages at once you can define here a maximum count of values that are published in one turn. After reaching this limit the MQTT Pusher will be forced to sleep for a short time before continuing.
|maxInflightMsgNum|Maximum number of messages that can be "inflight". This is a MQTT term and should match the broker's setting. Set to 0 for unlimited.
|maxQueuedMsgNum|Maximum number of MQTT messages (including "inflight") that should be queued. This is to limit the amount of memory that is used for buffering. Set to 0 for unlimited.
| verbosity | Level of detail in the logfile (dcdb.log). Set to a value between 5 (all log-messages, default) and 0 (only fatal messages). NOTE: level of verbosity for the command-line log can be set via the -v flag independently when invoking dcdbpusher.
......@@ -82,7 +82,7 @@ Please have a look at the provided `config/dcdbpusher.conf` example to get famil
| certificate | Provide the (path and) file which the HTTPS server should use as certificate.
| privateKey | Provide the (path and) file which should be used as corresponding private key for the certificate. If private key and certificate are stored in the same file one should nevertheless provide the path to the cert-file here again.
| dhFile | Provide the (path and) file where Diffie-Hellman parameters for the key exchange are stored.
| authkey | This struct is used to define authentication key tokens for the REST API. Within the struct, define which operations over the REST API are allowed for the token (e.g. PUTReq or GETReq). Each token must be unique.
| user | This struct is used to define username and password for users of the REST API, along with their respective allowed REST operations (i.e., GET, PUT, POST).
| | |
| plugins | In this section one can specify the plugins which should be used.
| plugin _name_ | The plugin name is used to build the corresponding lib-name (e.g. sysfs --> libdcdbplugin_sysfs.1.0)
......@@ -90,21 +90,23 @@ Please have a look at the provided `config/dcdbpusher.conf` example to get famil
| config | One can specify a separate config-file (including path to it) for the plugin to use. If not specified, dcdbpusher will look up pluginName.conf (e.g. sysfs.conf) in the same directory where dcdbpusher.conf is located.
| | |
Formats of the other sensor-specific config-files are explained in the corresponding [subsections](#ipmi). Example configuration-files can be found in the `config/` directory.
Formats of the other sensor-specific config-files are explained in the corresponding [subsections](#ipmi),
while example configuration-files can be found in the `config/` directory. An explanation of how to deploy Wintermute
data analytics plugins can be found in the corresponding readme document.
## REST API <a name="restApi"></a>
Dcdbpusher runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8000 on 127.0.0.1 but the address can be changed in the [`dcdbpusher.conf`](#globalConfiguration).
Dcdbpusher runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8000 on 127.0.0.1 but the address can be changed in [`dcdbpusher.conf`](#globalConfiguration).
A HTTPS request to dcdbpusher should have the following format: `[GET|PUT] host:port[ressource]?[queries]`.
Tables with allowed ressources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons(';'). For all requests (except /help) basic authentication credentials must be provided.
A HTTPS request to dcdbpusher should have the following format: `[GET|PUT|POST] host:port[resource]?[queries]`.
Tables with allowed resources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons(';'). For all requests (except /help) basic authentication credentials must be provided.
### List of ressources <a name="listOfRessources"></a>
### List of resources <a name="listOfResources"></a>
<table>
<tr>
<td colspan="2"><b>Ressource</b></td>
<td colspan="2"><b>Resource</b></td>
<td colspan="2">Description</td>
</tr>
<tr>
......
......@@ -3,14 +3,14 @@
### Table of contents
1. [Introduction](#introduction)
2. [grafanaserver](#grafanaserver)
1. [Configuration](#configuration)
2. [Rest API](#restApi)
1. [List of ressources](#listOfRessources)
2. [Examples](#restExamples)
1. [Configuration](#configuration)
2. [Rest API](#restApi)
1. [List of resources](#listOfResources)
2. [Examples](#restExamples)
3. [DCDB Grafana Plugin](#grafanaPlugin)
## Introduction <a name="introduction"></a>
DCDB (DataCenter DataBase) is a database to collect various (sensor-)values of a datacenter for further analysis.
DCDB (Data Center Data Base) is a database to collect various (sensor-)values of a data center for further analysis.
The grafanaserver is a REST HTTPS server that accepts and processes requests from the DCDB data source plugin of the Grafana web interface. It essentially resolves Grafana queries for visualizing timeserie data of systems monitored with DCDB.
# grafanaserver <a name="grafanaserver"></a>
......@@ -71,7 +71,7 @@ The configuration specifies various settings for grafanaserver in general. Pleas
| certificate | Provide the (path and) file which the HTTPS server should use as certificate.
| privateKey | Provide the (path and) file which should be used as corresponding private key for the certificate. If private key and certificate are stored in the same file one should nevertheless provide the path to the cert-file here again.
| dhFile | Provide the (path and) file where Diffie-Hellman parameters for the key exchange are stored.
| user | This struct is used to define username and password for users of the REST API, along with their respective allowed REST operations (i.e., GET, POST).
| user | This struct is used to define username and password for users of the REST API, along with their respective allowed REST operations (i.e., GET, PUT, POST).
| | |
An example configuration file can be found in the `config/` directory.
......@@ -79,16 +79,16 @@ An example configuration file can be found in the `config/` directory.
## REST API <a name="restApi"></a>
Grafanaserver runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8081 on 127.0.0.1 but the address can be changed in the [`grafana.conf`](#configuration).
Grafanaserver runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8081 on 127.0.0.1 but the address can be changed in [`grafana.conf`](#configuration).
Grafanaserver is designed to expect requests from a Grafana web interface, but it is of course possible to send generic HTTPS requests from different tools a well (e.g., curl). In this regard, a HTTPS request to grafanaserver should have the following format: `[GET|POST] host:port[ressource]?[queries]`.
Tables with allowed ressources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons(';'). For all requests, basic authentication credentials must be provided.
Grafanaserver is designed to expect requests from a Grafana web interface, but it is of course possible to send generic HTTPS requests from different tools a well (e.g., curl). In this regard, a HTTPS request to grafanaserver should have the following format: `[GET|PUT|POST] host:port[resource]?[queries]`.
Tables with allowed resources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons(';'). For all requests, basic authentication credentials must be provided.
### List of Ressources <a name="listOfRessources"></a>
### List of Resources <a name="listOfResources"></a>
<table>
<tr>
<td colspan="2"><b>Ressource</b></td>
<td colspan="2"><b>Resource</b></td>
<td colspan="2">Description</td>
</tr>
<tr>
......
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment