Commit 4d191371 authored by Alessio Netti's avatar Alessio Netti

Updates to documentation

parent 5a544c2d
......@@ -6,11 +6,11 @@ The *Data Center* *Data Base* (DCDB) is a modular, continuous and holistic monit
* _Collect_ _Agent_: The collect agent functions as intermediary between one storage backend and one or multiple pushers.
* _Pusher_: The main part for collecting data is the pusher framework. It allows to run arbitrary data collection plugins and pushes the data to the collect agent via MQTT messages.
Other features included by DCDB:
* libdcdb: a library to access the storage backend with external tools
* a set of tools leveraging libdcdb (e.g. a command line tool to query the storage backend)
* built in data analytics framework
* pusher plugins for a variety of typical HPC data sources
Other features included in DCDB:
* libdcdb: a library to access the storage backend with external tools.
* A set of tools leveraging libdcdb (e.g. a command line tool to query the storage backend).
* Built-in data analytics framework.
* Pusher plugins for a variety of typical HPC data sources.
## Build, install and run
......@@ -33,7 +33,7 @@ $ make depsinstall install
$ make depsinstall install doc
```
To run DCDB locally with the included default config files:
To run DCDB locally with the included default configuration files:
```bash
//Change to the DCDB directory you created during installation
......@@ -60,7 +60,7 @@ DCDB was created at Leibniz Supercomputing Centre (LRZ).
For questions and/or suggestions please contact info@dcdb.it
Copyright (C) 2011-2019 Leibniz Supercomputing Centre
Copyright (C) 2011-2020 Leibniz Supercomputing Centre
DCDB (except for the libdcdb parts) is licensed under the terms
and conditions of the GNU GENERAL PUBLIC LICENSE (Version 2 or later).
......
......@@ -16,7 +16,7 @@
5. [MQTT Topics](#mqttTopics)
6. [Pipelining Operators](#pipelining)
3. [Rest API](#restApi)
1. [List of ressources](#listOfRessources)
1. [List of resources](#listOfResources)
2. [Examples](#restExamples)
3. [Plugins](#plugins)
1. [Aggregator Plugin](#averagePlugin)
......@@ -29,7 +29,7 @@
7. [Tester Plugin](#testerPlugin)
4. [Sink Plugins](#sinkplugins)
1. [File Sink Plugin](#filesinkPlugin)
2. [Writing Plugins](#writingPlugins)
5. [Writing Plugins](#writingPlugins)
### Additional Resources
......@@ -528,13 +528,13 @@ Wintermute provides a REST API that can be used to perform various management op
API is functionally identical to that of DCDB Pusher, and is hosted at the same address. All requests that are targeted
at the data analytics framework must have a resource path starting with _/analytics_.
### List of ressources <a name="listOfRessources"></a>
### List of resources <a name="listOfResources"></a>
Prefix `/analytics` left out!
<table>
<tr>
<td colspan="2"><b>Ressource</b></td>
<td colspan="2"><b>Resource</b></td>
<td colspan="2">Description</td>
</tr>
<tr>
......@@ -962,7 +962,7 @@ Additionally, input sensors in sinks accept the following parameters:
|:----- |:----------- |
| path | The path to which the sensors's readings should be written. It is interpreted as described above for the _autoName_ attribute.
## Writing DCDB Analytics Plugins <a name="writingPlugins"></a>
# Writing Wintermute Plugins <a name="writingPlugins"></a>
Generating a DCDB Wintermute plugin requires implementing a _Operator_ and _Configurator_ class which contain all logic
tied to the specific plugin. Such classes should be derived from _OperatorTemplate_ and _OperatorConfiguratorTemplate_
respectively, which contain all plugin-agnostic configuration and runtime features. Please refer to the documentation
......
# Collect Agent
### Table of contents
1. [Introduction](#introduction)
2. [collectagent](#collectagent)
1. [Configuration](#configuration)
2. [Rest API](#restApi)
1. [List of resources](#listOfResources)
2. [Examples](#restExamples)
## Introduction <a name="introduction"></a>
DCDB (Data Center Data Base) is a database to collect sensor data associated to data center operation for further analysis.
The collectagent serves the role of data broker in the DCDB hierarchy.
# collectagent <a name="collectagent"></a>
The collectagent acts as an MQTT broker that subscribes to all topics, and has the task of forwarding the data sent by DCDB Pushers into an Apache Cassandra database.
It will try to batch database inserts, depending on the incoming MQTT packets, in order to optimize performance.
Build it by simply running
```bash
make
```
or alternatively use
```bash
make debug
```
within the `collectagent` directory to build a version which will print additional debug-information during runtime.
You can run a collectagent by executing
```bash
./collectagent path/to/configfile/
```
or run
```bash
./collectagent -h
```
to print the help-section of the collectagent.
The collectagent will check the given file-path for the global configuration file which has to be named `collectagent.conf`.
### Configuration <a name="configuration"></a>
The configuration specifies various general settings for the collectagent. Please have a look at the provided `config/collectagent.conf` example to get familiar with the file scheme. The different sections and values are explained in the following table:
| Value | Explanation |
|:----- |:----------- |
| global | Wrapper structure for the global values.
| threads | Specifies how many threads should be created for the local thread pool. This is used by the Wintermute plugins as well as the REST API. Default value of threads is 1.
| messageThreads | Number of threads to be used by the MQTT broker.
| messageSlots | Maximum number of connections allowed per MQTT thread.
| cacheInterval | Size (in seconds) of the local cache for each incoming sensor.
| verbosity | Level of detail in the log file. Set to a value between 5 (all log-messages, default) and 0 (only fatal messages). NOTE: level of verbosity for the command-line log can be set via the -v flag independently when invoking the collectagent.
| daemonize | Set to 'true' if the collectagent should run detached as daemon. Default is false.
| tempdir | One can specify a writeable directory where the collectagent can write its temporary and logging files to. Default is the current (' ./ ' ) directory.
| statistics | Boolean. If true, the collectagent will periodically print out statistic about data and insert rates.
| mqttListenAddresss | Address on which the MQTT broker must listen for incoming connections. Default is 127.0.0.1:1883.
| cleaningInterval | Periodic interval (in seconds) over which stale entries are purged from the sensor cache.
| | |
| cassandra | Wrapper for the Cassandra DB settings.
| address | Define address and port of the Cassandra DB server into which sensor data should be stored. Default is 127.0.0.1:9042.
| user | Username for Cassandra authentication. Default is empty.
| password | Password for Cassandra authentication. Default is empty.
| numThreadsIo | Number of I/O threads in the Cassandra driver. Default is 1.
| queueSizeIo | Size of the request queue for each I/O thread. Default is 4096.
| coreConnPerHost | Maximum number of connection per Cassandra database. Default is 1.
| ttl | Default *time to live* to be used for sensor data, if none is provided in the respective metadata.
| debugLog | Outputs verbose error messages if enabled. Default is false.
| | |
| restAPI | Bundles all values related to the RestAPI. See the corresponding [section](#restApi) for more information on supported functionality.
| address | Define (IP-)address and port where the REST API server should run on. Default is 127.0.0.1:8080.
| certificate | Provide the (path and) file which the HTTPS server should use as certificate.
| privateKey | Provide the (path and) file which should be used as corresponding private key for the certificate. If private key and certificate are stored in the same file one should nevertheless provide the path to the cert-file here again.
| dhFile | Provide the (path and) file where Diffie-Hellman parameters for the key exchange are stored.
| user | This struct is used to define username and password for users of the REST API, along with their respective allowed REST operations (i.e., GET, PUT, POST).
| | |
An example configuration file can be found in the `config/` directory. An explanation of how to deploy Wintermute
data analytics plugins can be found in the corresponding readme document.
## REST API <a name="restApi"></a>
The collectagent runs an HTTPS server which provides some functionality to be controlled over a RESTful API.
The API is by default hosted at port 8080 on 127.0.0.1 but the address can be changed in [`collectagent.conf`](#configuration).
An HTTPS request to the collectagent should have the following format: `[GET|PUT|POST] host:port[resource]?[queries]`.
Tables with allowed resources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons (';'). For all requests (except /help) basic authentication credentials must be provided.
### List of resources <a name="listOfResources"></a>
<table>
<tr>
<td colspan="2"><b>Resource</b></td>
<td colspan="2">Description</td>
</tr>
<tr>
<td>Query</td>
<td>Value</td>
<td>Opt.</td>
<td>Description</td>
</tr>
</table>
<table>
<tr>
<td colspan="2"><b>GET /help</b></td>
<td colspan="2">Return a cheatsheet of possible REST API endpoints.</td>
</tr>
<tr>
<td colspan="4">No queries.</td>
</tr>
</table>
<table>
<tr>
<td colspan="2"><b>GET /average</b></td>
<td colspan="2">Get the average of the last readings of a sensor in the local cache. Also allows access to analytics sensors.</td>
</tr>
<tr>
<td>sensor</td>
<td>All sensor names that are available.</td>
<td>No</td>
<td>Specify a sensor stored in the local cache.</td>
</tr>
<tr>
<td>interval</td>
<td>Number of seconds.</td>
<td>Yes</td>
<td>Use only readings more recent than (now - interval) for average calculation. Defaults to zero, i.e. all cached sensor readings are included in average calculation.</td>
</tr>
</table>
<table>
<tr>
<td colspan="2"><b>PUT /quit</b></td>
<td colspan="2">Exits the collectagent with a user-specified return code.</td>
</tr>
<tr>
<td>code</td>
<td>Return code.</td>
<td>Yes</td>
<td>Return code to be used when exiting.</td>
</tr>
</table>
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; Opt. = Optional
### Examples <a name="restExamples"></a>
Two examples for HTTPS requests (authentication credentials not shown):
```bash
GET https://localhost:8080/average?sensor=/clust06/node55/freq1;interval=15
```
```bash
PUT https://localhost:8080/quit?code=11
```
\ No newline at end of file
......@@ -5,7 +5,7 @@
2. [dcdbpusher](#dcdbpusher)
1. [Global Configuration](#globalConfiguration)
2. [Rest API](#restApi)
1. [List of ressources](#listOfRessources)
1. [List of resources](#listOfResources)
2. [Examples](#restExamples)
3. [MQTT topic](#mqttTopic)
3. [Plugins](#plugins)
......@@ -31,7 +31,7 @@ Harvesting of the data is task of the dcdbpusher.
# dcdbpusher <a name="dcdbpusher"></a>
This is a general MQTT pusher which sends values of various sensors to the DCDB-database.
It ships with plugins for BACnet, IPMI, PDU(proprietary Power Delivery Unit, but could be used as XML plugin), perfcounter, SNMP and sysFS.
It ships with plugins for BACnet, IPMI, PDU (proprietary Power Delivery Unit, but could be used as XML plugin), perfcounter, SNMP and sysFS, among others.
Build it by simply running
```bash
......@@ -68,8 +68,8 @@ Please have a look at the provided `config/dcdbpusher.conf` example to get famil
| mqttBroker | Define address and port of the MQTT-broker which collects the messages (sensor values) send by dcdbpusher.
| mqttprefix | To not rewrite a full MQTT-topic for every sensor one can specify here a consistent prefix.
| sensorpattern | pattern used to perform automatic sensor name publishing. See the corresponding [section](#autopublish) for more information.
| threads | Specify how many threads should be created to handle the sensors async. Default value of threads is 1. Note that the MQTTPusher always starts an extra thread. So the actual number of started threads is always one more than defined here. Specifying not enough threads can result in a delay for some sensors until they are read.
| maxMsgNum | To avoid publishing too many MQTT messages at once you can define here a maximum count of values that are published in one turn. After reaching this limit the MQTTPusher will be forced to sleep for a short time before continuing.
| threads | Specify how many threads should be created to handle the sensors asynchronously, as well as the Wintermute plugins. Default value of threads is 1. Note that the MQTT Pusher always starts an extra thread. So the actual number of started threads is always one more than defined here. Specifying not enough threads can result in a delay for some sensors until they are read.
| maxMsgNum | To avoid publishing too many MQTT messages at once you can define here a maximum count of values that are published in one turn. After reaching this limit the MQTT Pusher will be forced to sleep for a short time before continuing.
|maxInflightMsgNum|Maximum number of messages that can be "inflight". This is a MQTT term and should match the broker's setting. Set to 0 for unlimited.
|maxQueuedMsgNum|Maximum number of MQTT messages (including "inflight") that should be queued. This is to limit the amount of memory that is used for buffering. Set to 0 for unlimited.
| verbosity | Level of detail in the logfile (dcdb.log). Set to a value between 5 (all log-messages, default) and 0 (only fatal messages). NOTE: level of verbosity for the command-line log can be set via the -v flag independently when invoking dcdbpusher.
......@@ -82,7 +82,7 @@ Please have a look at the provided `config/dcdbpusher.conf` example to get famil
| certificate | Provide the (path and) file which the HTTPS server should use as certificate.
| privateKey | Provide the (path and) file which should be used as corresponding private key for the certificate. If private key and certificate are stored in the same file one should nevertheless provide the path to the cert-file here again.
| dhFile | Provide the (path and) file where Diffie-Hellman parameters for the key exchange are stored.
| authkey | This struct is used to define authentication key tokens for the REST API. Within the struct, define which operations over the REST API are allowed for the token (e.g. PUTReq or GETReq). Each token must be unique.
| user | This struct is used to define username and password for users of the REST API, along with their respective allowed REST operations (i.e., GET, PUT, POST).
| | |
| plugins | In this section one can specify the plugins which should be used.
| plugin _name_ | The plugin name is used to build the corresponding lib-name (e.g. sysfs --> libdcdbplugin_sysfs.1.0)
......@@ -90,21 +90,23 @@ Please have a look at the provided `config/dcdbpusher.conf` example to get famil
| config | One can specify a separate config-file (including path to it) for the plugin to use. If not specified, dcdbpusher will look up pluginName.conf (e.g. sysfs.conf) in the same directory where dcdbpusher.conf is located.
| | |
Formats of the other sensor-specific config-files are explained in the corresponding [subsections](#ipmi). Example configuration-files can be found in the `config/` directory.
Formats of the other sensor-specific config-files are explained in the corresponding [subsections](#ipmi),
while example configuration-files can be found in the `config/` directory. An explanation of how to deploy Wintermute
data analytics plugins can be found in the corresponding readme document.
## REST API <a name="restApi"></a>
Dcdbpusher runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8000 on 127.0.0.1 but the address can be changed in the [`dcdbpusher.conf`](#globalConfiguration).
Dcdbpusher runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8000 on 127.0.0.1 but the address can be changed in [`dcdbpusher.conf`](#globalConfiguration).
A HTTPS request to dcdbpusher should have the following format: `[GET|PUT] host:port[ressource]?[queries]`.
Tables with allowed ressources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons(';'). For all requests (except /help) basic authentication credentials must be provided.
A HTTPS request to dcdbpusher should have the following format: `[GET|PUT|POST] host:port[resource]?[queries]`.
Tables with allowed resources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons(';'). For all requests (except /help) basic authentication credentials must be provided.
### List of ressources <a name="listOfRessources"></a>
### List of resources <a name="listOfResources"></a>
<table>
<tr>
<td colspan="2"><b>Ressource</b></td>
<td colspan="2"><b>Resource</b></td>
<td colspan="2">Description</td>
</tr>
<tr>
......
......@@ -3,14 +3,14 @@
### Table of contents
1. [Introduction](#introduction)
2. [grafanaserver](#grafanaserver)
1. [Configuration](#configuration)
2. [Rest API](#restApi)
1. [List of ressources](#listOfRessources)
2. [Examples](#restExamples)
1. [Configuration](#configuration)
2. [Rest API](#restApi)
1. [List of resources](#listOfResources)
2. [Examples](#restExamples)
3. [DCDB Grafana Plugin](#grafanaPlugin)
## Introduction <a name="introduction"></a>
DCDB (DataCenter DataBase) is a database to collect various (sensor-)values of a datacenter for further analysis.
DCDB (Data Center Data Base) is a database to collect various (sensor-)values of a data center for further analysis.
The grafanaserver is a REST HTTPS server that accepts and processes requests from the DCDB data source plugin of the Grafana web interface. It essentially resolves Grafana queries for visualizing timeserie data of systems monitored with DCDB.
# grafanaserver <a name="grafanaserver"></a>
......@@ -71,7 +71,7 @@ The configuration specifies various settings for grafanaserver in general. Pleas
| certificate | Provide the (path and) file which the HTTPS server should use as certificate.
| privateKey | Provide the (path and) file which should be used as corresponding private key for the certificate. If private key and certificate are stored in the same file one should nevertheless provide the path to the cert-file here again.
| dhFile | Provide the (path and) file where Diffie-Hellman parameters for the key exchange are stored.
| user | This struct is used to define username and password for users of the REST API, along with their respective allowed REST operations (i.e., GET, POST).
| user | This struct is used to define username and password for users of the REST API, along with their respective allowed REST operations (i.e., GET, PUT, POST).
| | |
An example configuration file can be found in the `config/` directory.
......@@ -79,16 +79,16 @@ An example configuration file can be found in the `config/` directory.
## REST API <a name="restApi"></a>
Grafanaserver runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8081 on 127.0.0.1 but the address can be changed in the [`grafana.conf`](#configuration).
Grafanaserver runs a HTTPS server which provides some functionality to be controlled over a RESTful API. The API is by default hosted at port 8081 on 127.0.0.1 but the address can be changed in [`grafana.conf`](#configuration).
Grafanaserver is designed to expect requests from a Grafana web interface, but it is of course possible to send generic HTTPS requests from different tools a well (e.g., curl). In this regard, a HTTPS request to grafanaserver should have the following format: `[GET|POST] host:port[ressource]?[queries]`.
Tables with allowed ressources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons(';'). For all requests, basic authentication credentials must be provided.
Grafanaserver is designed to expect requests from a Grafana web interface, but it is of course possible to send generic HTTPS requests from different tools a well (e.g., curl). In this regard, a HTTPS request to grafanaserver should have the following format: `[GET|PUT|POST] host:port[resource]?[queries]`.
Tables with allowed resources sorted by REST methods can be found below. A query consists of a key-value pair of the format `key=value`. Multiple queries are separated by semicolons(';'). For all requests, basic authentication credentials must be provided.
### List of Ressources <a name="listOfRessources"></a>
### List of Resources <a name="listOfResources"></a>
<table>
<tr>
<td colspan="2"><b>Ressource</b></td>
<td colspan="2"><b>Resource</b></td>
<td colspan="2">Description</td>
</tr>
<tr>
......
# DCDB Tools
### Table of contents
1. [Introduction](#introduction)
2. [The dcdbconfig Tool](#dcdbconfig)
1. [Examples](#dcdbconfigexamples)
3. [The dcdbquery Tool](#dcdbquery)
1. [Examples](#dcdbqueryexamples)
4. [The dcdbslurmjob Tool](#dcdbslurmjob)
1. [Examples](#dcdbslurmjobexamples)
5. [Other Tools](#othertools)
# Introduction <a name="introduction"></a>
DCDB includes a series of tools to access stored sensor data as well as perform various types of actions. In this
document we review the main tools supplied with DCDB and their capabilities, along with several usage examples.
All
of the DCDB tools presented in the following interact directly with an Apache Cassandra instance, and do not require
communication with Pusher or Collect Agent components. Please remember that, to use any of the tools in the following,
the shell environment must be set properly using the supplied *dcdb.bash* script in the DCDB install directory:
```bash
$ source install/dcdb.bash
```
# The dcdbconfig Tool <a name="dcdbconfig"></a>
The *dcdbconfig* tool is used to perform database and metadata management actions. Its syntax is the following:
```bash
dcdbconfig [-h <host>] <command> [<arguments> ... ]
```
The *-h* argument specifies the hostname associated to the Apache Cassandra instance to which dcdbconfig should connect.
*\<command\>*, instead, indicates different types of actions and can be one of the following:
* *job*: allows to access actions that are job-related, e.g. to list the jobs stored in the database, visualize the ones
that are running, or show the details for a certain job ID. The insertion of the job data itself is performed via the
*dcdbslurmjob* tool.
* *sensor*: allows to perform several actions at the sensor level. For example, users can visualize and set the metadata
associated to a certain sensor name, list available sensors, or publish new ones.
* *db*: allows to perform certain database management actions, such as truncating obsolete data or manually inserting new data.
* *help*: visualizes a help message. More information for each of the commands above can be visualized by typing *help*
followed by the name of the specific command (e.g., *dcdbconfig help sensor*).
For each of the commands above a series of actions can be performed, which are specified with the subsequent arguments.
Some actions may further require additional arguments (e.g., the name of the sensor that the user wishes to visualize).
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; A *published* sensor in DCDB terminology is one that has its metadata correctly
populated in the Cassandra database. This process is usually handled transparently by Pushers that have the *-a* option
enabled. Publishing is not strictly necessary to store and query the sensor data itself.
## Examples <a name="dcdbconfigexamples"></a>
Visualizing the list of published sensors in the database:
```bash
dcdbconfig -h 127.0.0.1 sensor listpublic
```
Sample output, showing the mapping between the *public* sensor names (left) and their internal MQTT *topics*:
```
dcdbconfig 0.4.135-gb57cedf (libdcdb 0.4.135-gb57cedf)
/test/node1/AnonPages : /test/node1/AnonPages
/test/node1/Hugepagesize : /test/node1/Hugepagesize
/test/node1/MemFree : /test/node1/MemFree
/test/node1/col_nice : /test/node1/col_nice
/test/node1/col_system : /test/node1/col_system
/test/node1/col_user : /test/node1/col_user
```
---
Showing all available information for a specific sensor:
```bash
dcdbconfig -h 127.0.0.1 sensor show /test/node1/AnonPages
```
Sample output:
```
dcdbconfig 0.4.135-gb57cedf (libdcdb 0.4.135-gb57cedf)
Details for public sensor /test/node1/AnonPages:
Pattern: /test/node1/AnonPages
Unit: aaa
Scaling factor: 1
Operations: -avg10,-avg300,-avg3600
Interval: 500000000
TTL: 0
Sensor Properties:
```
---
Listing all jobs stored in the database that are currently running:
```bash
dcdbconfig -h 127.0.0.1 job running
```
Sample output:
```
dcdbconfig 0.4.135-gb57cedf (libdcdb 0.4.135-gb57cedf)
Job ID, User ID
800475,di726bwe
800499,ga526ppd
```
---
Showing the information associated to a specific job ID:
```bash
dcdbconfig -h 127.0.0.1 job show 800475
```
Sample output:
```
dcdbconfig 0.4.135-gb57cedf (libdcdb 0.4.135-gb57cedf)
Job ID: 800475
User ID: di726bwe
Start Time: 1595314022854049000
End Time: 0
Node List: /mpp2/r03/c04/s02/, /mpp2/r04/c04/s04/, /mpp2/r05/c05/s02/
```
# The dcdbquery Tool <a name="dcdbquery"></a>
The *dcdbquery* tool is used to perform database queries and fetch sensor data. Its syntax is the following:
```bash
dcdbquery -h <host> <Sensor 1> [<Sensor 2> ...] <Start> <End>
```
Like in dcdbconfig, the *-h* argument specifies the hostname associated to an Apache Cassandra instance. Then, the tool
accepts a list of sensor names to be queried (represented by the *\<Sensor\>* arguments) and a time range for the query
(represented by the *\<Start\>* and *\<End\>* arguments). The range of the query can either be supplied with a pair of 64-bit timestamps
(expressed in nanoseconds), or with two relative timestamps (e.g., *now-1h* as *Start* and *now* as *End*). Moreover, a
series of operations can be automatically performed over the returned set of sensor values. Operations can be specified
like in the following:
```bash
dcdbquery -h <host> operation(<Sensor 1>) [operation(<Sensor 2>) ...] <Start> <End>
```
The currently supported operations are *delta*, *delta_t*, *derivative* and *integral*.
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp;&ensp; Quoting of the dcdbquery command in the shell might be required to ensure proper parsing
when using sensor operations.
> NOTE 2 &ensp;&ensp;&ensp;&ensp;&ensp; By default, the dcdbquery tool only returns data for sensors that are correctly
published. This check can be bypassed by using the *-d* option.
## Examples <a name="dcdbqueryexamples"></a>
Querying sensor data in the last hour for a power sensor:
```bash
dcdbquery -h 127.0.0.1 /i01/r01/c01/s09/PKG_POWER now-1h now
```
Sample output:
```
dcdbquery 0.4.136-g233b476 (libdcdb 0.4.136-g233b476)
Sensor,Time,Value
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:28:00.000233000,135
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:30:00.000403000,198
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:32:00.000410000,190
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:34:00.000530000,84
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:36:00.002756000,259
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:38:00.000873000,239
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:40:00.000497000,96
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:42:00.000813000,256
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:44:00.002443000,256
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:46:00.002659000,261
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:48:00.001602000,259
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:50:00.001135000,260
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:52:00.000684000,264
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:54:00.001350000,261
/i01/r01/c01/s09/PKG_POWER,2020-08-04T09:56:00.001860000,255
```
---
Visualizing the delta_t for the successive readings of a DRAM power sensor:
```bash
dcdbquery -h 127.0.0.1 "delta_t(/sng/i01/r01/c01/s09/DRAM_POWER)" now-2h now
```
Sample output:
```
dcdbquery 0.4.136-g233b476 (libdcdb 0.4.136-g233b476)
Sensor,Time,Delta_t
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:14:00.000455000,
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:16:00.000426000,119999971000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:18:00.000420000,119999994000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:20:00.000405000,119999985000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:22:00.000385000,119999980000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:24:00.000403000,120000018000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:26:00.000458000,120000055000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:28:00.000438000,119999980000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:30:00.000403000,119999965000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:32:00.000430000,120000027000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:34:00.000467000,120000037000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:36:00.000491000,120000024000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:38:00.000664000,120000173000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:40:00.000649000,119999985000
/i01/r01/c01/s09/DRAM_POWER,2020-08-04T09:42:00.001142000,120000493000
```
---
Visualizing an unpublished sensor containing job-aggregated performance metrics:
```bash
dcdbquery -h 127.0.0.1 -d /job800745/FLOPS now-20m now
```
Sample output:
```
dcdbquery 0.4.136-g233b476 (libdcdb 0.4.136-g233b476)
Sensor,Time,Value
/job800745/FLOPS,2020-08-04T09:58:00.000138000,6509354081
/job800745/FLOPS,2020-08-04T10:00:00.000364000,32726063414
/job800745/FLOPS,2020-08-04T10:02:00.000109000,32114478744
/job800745/FLOPS,2020-08-04T10:04:00.000376000,32899551609
/job800745/FLOPS,2020-08-04T10:06:00.000384000,32769676466
/job800745/FLOPS,2020-08-04T10:08:00.000368000,31317069762
/job800745/FLOPS,2020-08-04T10:10:00.000460000,32422604906
/job800745/FLOPS,2020-08-04T10:12:00.000423000,32143846805
/job800745/FLOPS,2020-08-04T10:14:00.000709000,32193801528
/job800745/FLOPS,2020-08-04T10:16:00.000722000,32887149173
```
# The dcdbslurmjob Tool <a name="dcdbconfig"></a>
The *dcdbslurmjob* tool allows to manually insert job information in the Apache Cassandra database. Its syntax is the
following:
```bash
dcdbslurmjob [-b <host>] [-n <nodelist>] [-j <jobid>] [-i <userid>] start|stop
```
The job's information in terms of job ID, user ID and node list can be supplied via the *-j*, *-i* and *-n* arguments
respectively, the last of which being a comma-separated list. A timestamp (for the start or end of the job) can be
also optionally supplied with the *-t* argument: if not provided, this will default to the current time.
The dcdbslurmjob tool accepts two main commands, which are *start* and *stop*: the *start* command is invoked whenever
a new job must be inserted into the database (either at submission time, or at start time), whereas the *stop* command
is invoked to update the end time of an existing job with a specific ID.
There are two modes of transport for the dcdbslurmjob tool: if the *-b* argument is specified, the tool will attempt to
connect to the MQTT broker at the specified host, and transmit the job's information as an MQTT packet. On the other hand,
if the *-c* argument is used (optionally with the *-u* and *-p* arguments to specify a username and password respectively),
the tool will attempt to connect directly to the Apache Cassandra instance at the specified host.
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp;&ensp; While the dcdbslurmjob tool was designed primarily for HPC jobs orchestrated by
the SLURM resource manager, it is general enough to be applied to other environments as well.
## Examples <a name="dcdbslurmjobexamples"></a>
Inserting a new job into the database for the first time, by connecting directly to Cassandra:
```bash
dcdbslurmjob -c 127.0.0.1 -j 675222 -i po977yfg -n "/mpp3/r03/c05/s06/,/mpp3/r03/c05/s02/" start
```
Sample output:
```
dcdbslurmjob 0.4.135-gb57cedf
Connected to Cassandra server 127.0.0.1:9042
JOBID = 675222
USER = po977yfg
START = 1596543564693158000
NODELIST = /mpp3/r03/c05/s06/,/mpp3/r03/c05/s02/
SUBST =
NODES = /mpp3/r03/c05/s06/ /mpp3/r03/c05/s02/
```
---
Stopping the same job and setting its end timestamp:
```bash
dcdbslurmjob -c 127.0.0.1 -j 675222 stop
```
Sample output:
```
dcdbslurmjob 0.4.135-gb57cedf
Connected to Cassandra server 127.0.0.1:9042
JOBID = 675222
STOP = 1596543717613674000
```
# Other Tools <a name="othertools"></a>
Other tools, which are not covered in this document, are supplied with DCDB for specific use cases. These
are the following:
* *dcdbquerysum*: allows to compute integrals of multiple sensors over arbitrary time ranges.