README.md 34.2 KB
Newer Older
1
2
3
# DCDB Data Analytics Framework

### Table of contents
4
5
6
1. [Introduction](#introduction)
2. [DCDBAnalytics](#dcdbanalytics)
    1. [Global Configuration](#globalConfiguration)
7
    2. [Operators](#operators)
8
9
10
    	1. [The Sensor Tree](#sensorTree)
    	2. [The Unit System](#unitSystem)
    	3. [Operational Modes](#opModes)
11
		4. [Operator Configuration](#operatorConfiguration)
12
13
14
			1. [Configuration Syntax](#configSyntax)
			2. [Instantiating Units](#instantiatingUnits)
			3. [MQTT Topics](#mqttTopics)
15
16
			4. [Pipelining Operators](#pipelining)
			5. [Job Operators](#joboperators)
17
    3. [Rest API](#restApi)
Micha Mueller's avatar
Micha Mueller committed
18
19
        1. [List of ressources](#listOfRessources)
        2. [Examples](#restExamples)
20
3. [Plugins](#plugins)
Alessio Netti's avatar
Alessio Netti committed
21
22
	1. [Aggregator Plugin](#averagePlugin)
	2. [Job Aggregator Plugin](#jobaveragePlugin)
23
	3. [Regressor Plugin](#regressorPlugin)
24
25
26
27
	4. [Tester Plugin](#testerPlugin)
4. [Sink Plugins](#sinkplugins)
	1. [File Sink Plugin](#filesinkPlugin)
	2. [Writing Plugins](#writingPlugins)
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Introduction <a name="introduction"></a>
In this Readme we describe the DCDB Data Analytics framework, and all data abstractions that are associated with it. 

# DCDBAnalytics <a name="dcdbanalytics"></a>
The DCDBAnalytics framework is built on top of DCDB, and allows to perform data analytics based on sensor data
in a variety of ways. DCDBAnalytics can be deployed both in DCDBPusher and in DCDBCollectAgent, with some minor
differences:

* **DCDBPusher**: only sensor data that is sampled locally and that is contained within the sensor cache can be used for
data analytics. However, this is the preferable way to deploy simple models on a large-scale, as all computation is
performed within compute nodes, dramatically increasing scalability;
* **DCDBCollectAgent**: all available sensor data, in the local cache and in the Cassandra database, can be used for data
analytics. This approach is preferable for models that require data from multiple sources at once 
42
(e.g., clustering-based anomaly detection), or for models that are deployed in [on-demand](#operatorConfiguration) mode.
43
44
45
46
47
48
49
50

## Global Configuration <a name="globalConfiguration"></a>
DCDBAnalytics shares the same configuration structure as DCDBPusher and DCDBCollectAgent, using a global.conf configuration file. 
All output sensors of the frameworks are therefore affected by configuration parameters described in the global Readme. 
Additional parameters specific to this framework are the following:

| Value | Explanation |
|:----- |:----------- |
Alessio Netti's avatar
Alessio Netti committed
51
| **analytics** | Wrapper structure for the data analytics-specific values.
52
| hierarchy | Space-separated sequence of regular expressions used to infer the local (DCDBPusher) or global (DCDBCollectAgent) sensor hierarchy. This parameter should be wrapped in quotes to ensure proper parsing. See the Sensor Tree [section](#sensorTree) for more details.
Alessio Netti's avatar
Alessio Netti committed
53
54
55
| filter | Regular expression used to filter the set of sensors in the sensor tree. Everything that matches is included, the rest is discarded.
| jobFilter | Regular expression used to filter the jobs processed by job operators. The expression is applied to the first node of the job's nodelist. If a match is found the job is processed, otherwise it is discarded. This behavior can be changed at the plugin level.
| **operatorPlugins** | Block containing the specification of all data analytics plugin to be instantiated.
56
| plugin _name_ | The plugin name is used to build the corresponding lib-name (e.g. average --> libdcdboperator_average.1.0)
57
58
59
60
| path | Specify the path where the plugin (the shared library) is located. If left empty, DCDB will look in the default lib-directories (usr/lib and friends) for the plugin file.
| config | One can specify a separate config-file (including path to it) for the plugin to use. If not specified, DCDB will look up pluginName.conf (e.g. average.conf) in the same directory where global.conf is located.
| | |

61
62
63
## Operators <a name="operators"></a>
Operators are the basic building block in DCDBAnalytics. A Operator is instantiated within a plugin, performs a specific
task and acts on sets of inputs and outputs called _units_. Operators are functionally equivalent to _sensor groups_
64
in DCDBPusher, but instead of sampling data, they process such data and output new sensors. Some high-level examples
65
of operators are the following:
66

67
68
* An operator that performs time series regression on a particular input sensor, and outputs its prediction;
* An operator that aggregates a series of input sensors, builds feature vectors, and performs machine 
69
learning-based tasks using a supervised model;
70
* An operator that performs clustering-based anomaly detection by using different sets of inputs associated to different
71
compute nodes;
72
* An operator that outputs statistical features related to the time series of a certain input sensor.
73
74

### The Sensor Tree <a name="sensorTree"></a>
75
Before diving into the configuration and instantiation of operators, we introduce the concept of _sensor tree_. A  sensor
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
tree is simply a data structure expressing the hierarchy of sensors that are being sampled; internal nodes express
hierarchical entities (e.g. clusters, racks, nodes, cpus), whereas leaf nodes express actual sensors. In DCDBPusher, 
a sensor tree refers only to the local hierarchy, while in DCDBCollectAgent it can capture the hierarchy of the entire
system being sampled.

A sensor tree is built at initialization time of DCDBAnalytics, and is implemented in the _SensorNavigator_ class. 
Its construction is regulated by the _hierarchy_ global configuration parameter, which can be for example the following:

```
rack\d{2}. node\d{2}. cpu\d{2}. 
``` 

Given the above example hierarchy string, we are enforcing the target sensor tree to have three levels, the topmost of
which expresses the racks to which sensors belong, and the lowest one the cpu core (if any). Such string could be used,
for example to build a sensor tree starting from the following set of sensor names:

```
rack00.status
rack00.node05.MemFree
rack00.node05.energy
rack00.node05.temp
rack00.node05.cpu00.col_user
rack00.node05.cpu00.instr
rack00.node05.cpu00.branch-misses
rack00.node05.cpu01.col_user
rack00.node05.cpu01.instr
rack00.node05.cpu01.branch-misses
rack02.status
rack02.node03.MemFree
rack02.node03.energy
rack02.node03.temp
rack02.node03.cpu00.col_user
rack02.node03.cpu00.instr
rack02.node03.cpu00.branch-misses
rack02.node03.cpu01.col_user
rack02.node03.cpu01.instr
rack02.node03.cpu01.branch-misses
``` 

Each sensor name is interpreted as a path within the sensor tree. Therefore, the _instr_ and _branch-misses_ sensors
will be placed as leaf nodes in the deepest level of the tree, as children of the respective cpu node they belong to.
Such cpu nodes will be in turn children of the nodes they belong to, and so on.

The generated sensor tree can then be used to navigate the sensor hierarchy, and perform actions such as _retrieving
all sensors belonging to a certain node, to a neighbor of a certain node, or to the rack a certain node belongs to_.
Please refer to the documentation of the _SensorNavigator_ class for more details.

> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; If no hierarchy string has been specified in the configuration, the tree is built automatically by assuming that each
dot-separated part of the sensor name expresses a level in the hierarchy. The total depth of the tree is thus determined
at runtime as well.

> NOTE 2 &ensp;&ensp;&ensp; Sensor trees are always built from the names of sensors _as they are published_. Therefore,
please make sure to use the _-a_ option in DCDBPusher appropriately, to build sensor names that express the desired hierarchy.


### The Unit System <a name="unitSystem"></a>
132
Each operator operates on one or more _units_. A unit represents an abstract (or physical) entity in the current system that
133
134
135
136
137
138
139
140
141
is the target of analysis. A unit could be, for example, a rack, a node within a rack, a CPU within a node or an entire HPC system.
Units are identified by three components:

* **Name**: The name of this unit, that corresponds to the entity it represents. For example, _rack02.node03._ or _rack00.node05.cpu01._ could be unit names. A unit must always correspond to an existing internal node in the current sensor tree;
* **Input**: The set of sensors that constitute the input for analysis conducted on this unit. The sensors must share a hierarchical relationship with the unit: that is, they can either belong to the node represented by this unit, to its subtree, or to one of its ancestors; 
* **Output**: The set of output sensors that are produced from any analysis conducted on this unit. The output sensors are always directly associated with the node represented by the unit.

Units are a way to define _patterns_ in the sensor tree and retrieve sensors that are associated to each other by a 
hierarchical relationship. See the configuration [section](#instantiatingUnits) for more details on how to create
142
templates in order to define units suitable for operators.
143
144

### Operational Modes <a name="opModes"></a>
145
Operators can operate in two different modes:
146

147
148
149
* **Streaming**: streaming operators perform data analytics online and autonomously, processing incoming sensor data at regular intervals.
The units of streaming operators are completely resolved and instantiated at configuration time. The type of output of streaming
operators is identical to that of _sensors_ in DCDBPusher, which are pushed to DCDBCollectAgent and finally to the Cassandra database,
150
resulting in a time series representation;
151
152
153
* **On-demand**: on-demand operators do not perform data analytics autonomously, but only when queried by users. Unlike
for streaming operators, the units of on-demand operators are not instantiated at configuration, but only when a query is performed. When 
such an event occurs, the operator verifies that the queried unit belongs to its _unit domain_, and then instantiates it,
154
resolving its inputs and outputs. Then, the unit is stored in a local cache for future re-use. The outputs of a on-demand
155
operator are exposed through the REST API, and are never pushed to the Cassandra database.
156

157
Use of streaming operators is advised when a time series-like output is required, whereas on-demand operators are effective
158
when data is required at specific times and for specific purposes, and when the unit domain's size makes the use of streaming
159
operators unfeasible.
160

161
162
### Operator Configuration <a name="operatorConfiguration"></a>
Here we describe how to configure and instantiate operators in DCDBAnalytics. The configuration scheme is very similar
163
to that of _sensor groups_ in DCDBPusher, and a _global_ configuration block can be defined in each plugin configuration
164
file. The following is instead a list of configuration parameters that are available for the operators themselves:
165
166
167

| Value | Explanation |
|:----- |:----------- |
168
169
170
| default | Name of the template that must be used to configure this operator.
| interval | Specifies how often the operator will be invoked to perform computations, and thus the sampling interval of its output sensors. Only used for operators in _streaming_ mode.
| relaxed | If set to _true_, the units of this operator will be instantiated even if some of the respective input sensors do not exist.
171
| delay | Delay in milliseconds to be applied to the interval of the operator. This parameter can be used to tune how operator pipelines work, ensuring that the next computation stage is started only after the previous one has finished.
172
| unitCacheLimit | Defines the maximum size of the unit cache that is used in the on-demand and job modes. Default is 1000.
173
174
| minValues |   Minimum number of readings that need to be stored in output sensors before these are pushed as MQTT messages. Only used for operators in _streaming_ mode.
| mqttPart |    Part of the MQTT topic associated to this operator. Only used when the Unit system is not employed (see this [section](#mqttTopics)).
175
| sync | If set to _true_, computation will be performed at time intervals synchronized with sensor readings.
Alessio Netti's avatar
Alessio Netti committed
176
| disabled | If set to _true_, the operator will be instantiated but will not be started and will not be available for queries.
177
178
179
180
| duplicate | 	If set to _false_, only one operator object will be instantiated. Such operator will perform computation over all units that are instantiated, at every interval, sequentially. If set to _true_, the operator object will be duplicated such that each copy will have one unit associated to it. This allows to exploit parallelism between units, but results in separate models to avoid race conditions.
| streaming |	If set to _true_, the operator will operate in _streaming_ mode, pushing output sensors regularly. If set to _false_, the operator will instead operate in _on-demand_ mode.
| input | Block of input sensors that must be used to instantiate the units of this operator. These can both be a list of strings, or fully-qualified _Sensor_ blocks containing specific attributes (see DCDBPusher Readme).
| output | Block of output sensors that will be associated to this operator. These must be _Sensor_ blocks containing valid MQTT suffixes. Note that the number of output sensors is usually fixed depending on the type of operator.
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
| | |

#### Configuration Syntax <a name="configSyntax"></a>
In the following we show a sample configuration block for the _Average_ plugin. For the full version, please refer to the
default configuration file in the _config_ directory:

```
template_average def1 {
interval	1000
minValues	3
duplicate 	false
streaming	true
}

average avg1 {
default     def1
197
mqttPart    /avg1
198
199
200
201
202
203
204
205

	input {
		sensor col_user
		sensor MemFree
	}

	output {
		sensor sum {
206
			mqttsuffix  /sum
207
208
209
		}

		sensor max {
210
			mqttsuffix  /max
211
212
213
		}

		sensor avg {
214
			mqttsuffix  /avg
215
216
217
218
219
220
		}
	}
}
``` 

The configuration shown above uses a template _def1_ for some configuration parameters, which are then applied to the
221
_avg1_ operator. This operator takes the _col_user_ and _MemFree_ sensors as input (which must be available under this name),
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
 and outputs _sum_, _max_, and _avg_ sensors. In this configuration, the Unit system and sensor hierarchy are not used, 
 and therefore only one generic unit (called the _root_ unit) will be instantiated.

#### Instantiating Units <a name="instantiatingUnits"></a>
Here we propose once again the configuration discussed above, this time making use of the Unit system to abstract from
the specific system being used and simplify configuration. The adjusted configuration block is the following: 

```
template_average def1 {
interval	1000
minValues	3
duplicate 	false
streaming	true
}

average avg1 {
default     def1
239
mqttPart    /avg1
240
241
242
243
244
245
246

	input {
		sensor "<bottomup>col_user"
		sensor "<bottomup 1>MemFree"
	}

	output {
Alessio Netti's avatar
Alessio Netti committed
247
		sensor "<bottomup, filter cpu00>sum" {
248
			mqttsuffix  /sum
249
250
		}

Alessio Netti's avatar
Alessio Netti committed
251
		sensor "<bottomup, filter cpu00>max" {
252
			mqttsuffix  /max
253
254
		}

Alessio Netti's avatar
Alessio Netti committed
255
		sensor "<bottomup, filter cpu00>avg" {
256
			mqttsuffix  /avg
257
258
259
260
261
262
		}
	}
}
``` 

In each sensor declaration, the _< >_ block is a placeholder that will be replaced with the name of the units that will
263
be associated to the operator, thus resolving the sensor names. Such block allows to navigate the current sensor tree,
264
265
266
267
268
269
270
271
272
273
274
275
276
277
and select nodes that will constitute the units. Its syntax is the following:

```
< bottomup|topdown X, filter Y >SENSORNAME 
``` 

The first section specified the _level_ in the sensor tree at which nodes must be selected. _bottomup X_ and _topdown X_
respectively mean _"search X levels up from the deepest level in the sensor tree"_, and _"search X levels down from the 
topmost level in the sensor tree"_. The _X_ numerical value can be omitted as well.

The second section, on the other hand, allows to search the sensor tree _horizontally_. Within the level specified in the
first section of the configuration block, only the nodes whose names match with the regular expression Y will be selected.
This way, we can navigate the current sensor tree both vertically and horizontally, and easily instantiate units starting 
from nodes in the tree. The set of nodes in the current sensor tree that match with the specified configuration block is
278
defined as the _unit domain_ of the operator.
279
280
281

The configuration algorithm then works in two steps:

282
1. The _output_ block of the operator is read, and its unit domain is determined; this implies that all sensors in the 
283
284
285
output block must share the same _< >_ block, and therefore match the same unit domain;
2. For each unit in the domain, its input sensors are identified. We start from the _unit_ node in the sensor tree, and 
navigate to the corresponding sensor node according to its _< >_ block, which identifies its level in the tree. Each 
286
unit, once its inputs and outputs are defined, is then added to the operator.
287
288

According to the sensor tree built in the previous [section](#sensorTree), the configuration above would result in
289
an operator with the following set of units:
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318

```
rack00.node05.cpu00. {
	Inputs {
		rack00.node05.cpu00.col_user
		rack00.node05.MemFree
	}
	
	Outputs {
		rack00.node05.cpu00.sum
		rack00.node05.cpu00.max
		rack00.node05.cpu00.avg
	}
}

rack02.node03.cpu00. {
	Inputs {
		rack02.node03.cpu00.col_user
		rack02.node03.MemFree
	}
                     	
	Outputs {
		rack02.node03.cpu00.sum
		rack02.node03.cpu00.max
		rack02.node03.cpu00.avg
	}
}
``` 

Alessio Netti's avatar
Alessio Netti committed
319
320
321
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; In order to get units that operate at the _node_ level, the output sensors in the
configuration discussed here should have a unit block in the form of < bottomup 1, filter cpu00 >.

322
#### MQTT Topics <a name="mqttTopics"></a>
323
The MQTT topics associated to output sensors of a certain operator are constructed in different ways depending
324
325
326
327
on the unit they belong to:

* **Root unit**: if the output sensors belong to the _root_ unit, that is, they do not belong to any level in the sensor
hierarchy and are uniquely defined, the respective topics are constructed like in DCDBPusher sensors, by concatenating
328
the MQTT prefix, operator part and sensor suffix that are defined;
Alessio Netti's avatar
Alessio Netti committed
329
330
* **Job unit**: if the output sensors belong to a _job_ unit in a job operator (see below), the MQTT topic is constructed
by concatenating the MQTT prefix, the operator part, a job suffix (e.g., /job1334) and finally the sensor suffix;
331
332
* **Any other unit**: if the output sensor belongs to any other unit in the sensor tree, its MQTT topic is constructed
by concatenating the MQTT prefix associated to the unit (which is defined as _the portion of the MQTT topic shared by all sensors
Alessio Netti's avatar
Alessio Netti committed
333
belonging to such unit_) and the sensor suffix.
334

335
#### Pipelining Operators <a name="pipelining"></a>
336

337
338
339
The inputs and outputs of streaming operators can be chained so as to form a processing pipeline. To enable this, users
need to configure operators by enabling the _relaxed_ configuration parameter, and by selecting as input the output sensors
of other operators. This is necessary as the operators are instantiated sequentially at startup, and
340
341
the framework cannot infer the correct order of initialization so as to resolve all dependencies transparently.

342
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; This feature is not supported when using operators in _on demand_ mode.
343

344
#### Job Operators <a name="joboperators"></a>
Alessio Netti's avatar
Alessio Netti committed
345

346
_Job Operators_ are a class of operators which act on job-specific data. Such data is structured in _job units_. These
Alessio Netti's avatar
Alessio Netti committed
347
348
349
350
351
352
work similarly to ordinary units, with the difference that they are arranged hierarchically as follows:

* The top unit is associated to the job itself and contains all of the required output sensors;
* One sub-unit for each node on which the job was running is allocated. Each of these sub-units contains all of the input
sensors that are required at configuration time.

353
The computation algorithms driving job operators can then navigate freely this hierarchical unit design according to
Alessio Netti's avatar
Alessio Netti committed
354
355
356
their specific needs. Since all of the sub-units are tied to the nodes on which the job was running, the output sensors
specified in the configuration must also be at the node level in the unit system, such that unit resolution is performed correctly.

357
Job operators also support the _streaming_ and _on demand_ modes, which work like the following:
Alessio Netti's avatar
Alessio Netti committed
358

359
* In **streaming** mode, the job operator will retrieve the list of jobs that were running in the time interval starting
Alessio Netti's avatar
Alessio Netti committed
360
361
362
from the last computation to the present; it will then build one job unit for each of them, and subsequently perform computation;
* In **on demand** mode, users can query a specific job id, for which a job unit is built and computation is performed.

363
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; The _duplicate_ setting does not affect job operators.
Alessio Netti's avatar
Alessio Netti committed
364

365
366
367
368
369
## Rest API <a name="restApi"></a>
DCDBAnalytics provides a REST API that can be used to perform various management operations on the framework. The 
API is functionally identical to that of DCDBPusher, and is hosted at the same address. All requests that are targeted
at the data analytics framework must have a resource path starting with _/analytics_.

Micha Mueller's avatar
Micha Mueller committed
370
371
### List of ressources <a name="listOfRessources"></a>

Micha Mueller's avatar
Micha Mueller committed
372
Prefix `/analytics` left out!
Micha Mueller's avatar
Micha Mueller committed
373
374
375

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
376
    <td colspan="2"><b>Ressource</b></td>
Micha Mueller's avatar
Micha Mueller committed
377
378
379
380
381
382
383
384
385
386
387
388
    <td colspan="2">Description</td>
  </tr>
  <tr>
  	<td>Query</td>
  	<td>Value</td>
  	<td>Opt.</td>
  	<td>Description</td>
  </tr>
</table>

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
389
    <td colspan="2"><b>GET /help</b></td>
Micha Mueller's avatar
Micha Mueller committed
390
391
392
393
394
395
396
397
398
    <td colspan="2">Return a cheatsheet of possible analytics REST API endpoints.</td>
  </tr>
  <tr>
  	<td colspan="4">No queries.</td>
  </tr>
</table>

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
399
    <td colspan="2"><b>GET /plugins</b></td>
Micha Mueller's avatar
Micha Mueller committed
400
401
402
403
404
    <td colspan="2">List all currently loaded data analytic plugins.</td>
  </tr>
  <tr>
  	<td>json</td>
  	<td>"true"</td>
Micha Mueller's avatar
Micha Mueller committed
405
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
406
407
408
409
410
411
  	<td>Format response as json.</td>
  </tr>
</table>

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
412
    <td colspan="2"><b>GET /sensors</b></td>
Micha Mueller's avatar
Micha Mueller committed
413
414
415
416
    <td colspan="2">List all sensors of a specific plugin.</td>
  </tr>
  <tr>
  	<td>plugin</td>
417
  	<td>All operator plugin names.</td>
Micha Mueller's avatar
Micha Mueller committed
418
  	<td>No</td>
Micha Mueller's avatar
Micha Mueller committed
419
420
421
  	<td>Specify the plugin.</td>
  </tr>
  <tr>
422
423
  	<td>operator</td>
  	<td>All operators of a plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
424
  	<td>Yes</td>
425
  	<td>Restrict sensor list to an operator.</td>
Micha Mueller's avatar
Micha Mueller committed
426
427
428
429
  </tr>
  <tr>
  	<td>json</td>
  	<td>"true"</td>
Micha Mueller's avatar
Micha Mueller committed
430
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
431
432
433
434
435
436
  	<td>Format response as json.</td>
  </tr>
</table>

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
437
    <td colspan="2"><b>GET /units</b></td>
Micha Mueller's avatar
Micha Mueller committed
438
439
440
441
    <td colspan="2">List all units of a specific plugin.</td>
  </tr>
  <tr>
  	<td>plugin</td>
442
  	<td>All operator plugin names.</td>
Micha Mueller's avatar
Micha Mueller committed
443
  	<td>No</td>
Micha Mueller's avatar
Micha Mueller committed
444
445
446
  	<td>Specify the plugin.</td>
  </tr>
  <tr>
447
448
  	<td>operator</td>
  	<td>All operators of a plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
449
  	<td>Yes</td>
450
  	<td>Restrict unit list to an operator.</td>
Micha Mueller's avatar
Micha Mueller committed
451
452
453
454
  </tr>
  <tr>
  	<td>json</td>
  	<td>"true"</td>
Micha Mueller's avatar
Micha Mueller committed
455
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
456
457
458
459
460
461
  	<td>Format response as json.</td>
  </tr>
</table>

<table>
  <tr>
462
463
    <td colspan="2"><b>GET /operators</b></td>
    <td colspan="2">List all operators of a specific plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
464
465
466
  </tr>
  <tr>
  	<td>plugin</td>
467
  	<td>All operator plugin names.</td>
Micha Mueller's avatar
Micha Mueller committed
468
  	<td>No</td>
Micha Mueller's avatar
Micha Mueller committed
469
470
471
472
473
  	<td>Specify the plugin.</td>
  </tr>
  <tr>
  	<td>json</td>
  	<td>"true"</td>
Micha Mueller's avatar
Micha Mueller committed
474
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
475
476
477
478
479
480
  	<td>Format response as json.</td>
  </tr>
</table>

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
481
    <td colspan="2"><b>PUT /start</b></td>
482
    <td colspan="2">Start all or only a specific plugin. Or only start a specific streaming operator within a specific plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
483
484
485
486
  </tr>
  <tr>
  	<td>plugin</td>
  	<td>All plugin names.</td>
Micha Mueller's avatar
Micha Mueller committed
487
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
488
489
490
  	<td>Specify the plugin.</td>
  </tr>
  <tr>
491
492
  	<td>operator</td>
  	<td>All operator names of a plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
493
  	<td>Yes</td>
494
  	<td>Only start the specified operator. Requires a plugin to be specified. Limited to streaming operators.</td>
Micha Mueller's avatar
Micha Mueller committed
495
496
497
498
499
  </tr>
</table>

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
500
    <td colspan="2"><b>PUT /stop</b></td>
501
    <td colspan="2">Stop all or only a specific plugin. Or only stop a specific streaming operator within a specific plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
502
503
504
505
  </tr>
  <tr>
  	<td>plugin</td>
  	<td>All plugin names.</td>
Micha Mueller's avatar
Micha Mueller committed
506
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
507
508
509
  	<td>Specify the plugin.</td>
  </tr>
  <tr>
510
511
  	<td>operator</td>
  	<td>All operator names of a plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
512
  	<td>Yes</td>
513
  	<td>Only stop the specified operator. Requires a plugin to be specified. Limited to streaming operators.</td>
Micha Mueller's avatar
Micha Mueller committed
514
515
516
517
518
  </tr>
</table>

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
519
    <td colspan="2"><b>PUT /reload</b></td>
520
    <td colspan="2">Reload configuration and initialization of all or only a specific operator plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
521
522
523
524
  </tr>
  <tr>
  	<td>plugin</td>
  	<td>All plugin names.</td>
Micha Mueller's avatar
Micha Mueller committed
525
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
526
527
528
529
530
531
  	<td>Reload only the specified plugin.</td>
  </tr>
</table>

<table>
  <tr>
Micha Mueller's avatar
Micha Mueller committed
532
    <td colspan="2"><b>PUT /compute</b></td>
533
    <td colspan="2">Query the given operator for a certain input unit. Intended for "on-demand" operators, but works with "streaming" operators as well.</td>
Micha Mueller's avatar
Micha Mueller committed
534
535
536
537
  </tr>
  <tr>
  	<td>plugin</td>
  	<td>All plugin names.</td>
Micha Mueller's avatar
Micha Mueller committed
538
  	<td>No</td>
Micha Mueller's avatar
Micha Mueller committed
539
540
541
  	<td>Specify the plugin.</td>
  </tr>
  <tr>
542
543
  	<td>operator</td>
  	<td>All operator names of a plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
544
  	<td>No</td>
545
  	<td>Specify the operator within the plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
546
547
548
549
  </tr>
  <tr>
  	<td>unit</td>
  	<td>All units of a plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
550
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
551
552
553
554
555
  	<td>Select the target unit. Defaults to the root unit if not specified.</td>
  </tr>
  <tr>
  	<td>json</td>
  	<td>"true"</td>
Micha Mueller's avatar
Micha Mueller committed
556
  	<td>Yes</td>
Micha Mueller's avatar
Micha Mueller committed
557
558
559
560
561
562
  	<td>Format response as json.</td>
  </tr>
</table>

<table>
  <tr>
563
564
    <td colspan="2"><b>PUT /operator</b></td>
    <td colspan="2">Perform a custom REST PUT action defined at operator level. See operator plugin documenation for such actions.</td>
Micha Mueller's avatar
Micha Mueller committed
565
566
567
568
  </tr>
  <tr>
  	<td>plugin</td>
  	<td>All plugin names.</td>
Micha Mueller's avatar
Micha Mueller committed
569
  	<td>No</td>
Micha Mueller's avatar
Micha Mueller committed
570
571
572
573
  	<td>Specify the plugin.</td>
  </tr>
  <tr>
  	<td>action</td>
574
  	<td>See operator plugin documentation.</td>
Micha Mueller's avatar
Micha Mueller committed
575
  	<td>No</td>
Micha Mueller's avatar
Micha Mueller committed
576
577
578
  	<td>Select custom action.</td>
  </tr>
  <tr>
579
580
  	<td>operator</td>
  	<td>All operators of a plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
581
  	<td>Yes</td>
582
  	<td>Specify the operator within the plugin.</td>
Micha Mueller's avatar
Micha Mueller committed
583
584
585
586
587
588
  </tr>
  <tr>
  	<td colspan="4">Custom action may require or allow for more queries!</td>
  </tr>
</table>

Micha Mueller's avatar
Micha Mueller committed
589
590
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; Opt. = Optional

591
> NOTE 2 &ensp;&ensp;&ensp;&ensp;&ensp; The value of operator output sensors can be retrieved with the _compute_ resource, or with the [plugin]/[sensor]/avg resource defined in the DCDBPusher REST API.
Micha Mueller's avatar
Micha Mueller committed
592

593
> NOTE 3 &ensp;&ensp;&ensp;&ensp;&ensp; Developers can integrate their custom REST API resources that are plugin-specific, by implementing the _REST_ method in _OperatorTemplate_. To know more about plugin-specific resources, please refer to the respective documentation. 
594
595
596
597

### Rest Examples <a name="restExamples"></a>
In the following are some examples of REST requests over HTTPS:

598
* Listing the units associated to the _avgoperator1_ operator in the _average_ plugin:
599
```bash
600
GET https://localhost:8000/analytics/units?plugin=average;operator=avgOperator1
601
```
602
* Listing the output sensors associated to all operators in the _average_ plugin:
603
```bash
Micha Mueller's avatar
Micha Mueller committed
604
GET https://localhost:8000/analytics/sensors?plugin=average;
605
606
607
```
* Reloading the _average_ plugin:
```bash
Micha Mueller's avatar
Micha Mueller committed
608
PUT https://localhost:8000/analytics/reload?plugin=average
609
```
610
* Stopping the _avgOperator1_ operator in the _average_ plugin:
611
```bash
612
PUT https://localhost:8000/analytics/stop?plugin=average;operator=avgOperator1
613
```
614
* Performing a query for unit _node00.cpu03._ to the _avgOperator1_ operator in the _average_ plugin:
615
```bash
616
PUT https://localhost:8000/analytics/compute?plugin=average;operator=avgOperator1;unit=node00.cpu03
617
618
```

619
620
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; The analytics RestAPI requires authentication credentials as well.

621
622
623
# Plugins <a name="plugins"></a>
Here we describe available plugins in DCDBAnalytics, and how to configure them.

Alessio Netti's avatar
Alessio Netti committed
624
625
## Aggregator Plugin <a name="averagePlugin"></a>
The _Aggregator_ plugin implements simple data processing algorithms. Specifically, this plugin allows to perform basic
626
aggregation operations over a set of input sensors, which are then written as output.
Alessio Netti's avatar
Alessio Netti committed
627
The configuration parameters specific to the _Aggregator_ plugin are the following:
628
629
630
631

| Value | Explanation |
|:----- |:----------- |
| window | Length in milliseconds of the time window that is used to retrieve recent readings for the input sensors, starting from the latest one.
632

633
Additionally, output sensors in operators of the Aggregator plugin accept the following parameters:
634
635
636

| Value | Explanation |
|:----- |:----------- |
637
| operation | Operation to be performed over the input sensors. Can be "sum", "average", "maximum", "minimum", "std", "percentiles" or "observations".
638
| percentile |  Specific percentile to be computed when using the "percentiles" operation. Can be an integer in the (0,100) range.
639
640
| relative | If true, the _relative_ query mode will be used. Otherwise the _absolute_ mode is used.

641

Alessio Netti's avatar
Alessio Netti committed
642
643
644
645
## Job Aggregator Plugin <a name="jobaveragePlugin"></a>

The _Job Aggregator_ plugin offers the same functionality as the _Aggregator_ plugin, but on a per-job basis. As such,
it performs aggregation of the specified input sensors across all nodes on which each job is running. Please refer
646
to the corresponding [section](#joboperators) for more details.
Alessio Netti's avatar
Alessio Netti committed
647

648
649
> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; The Job Aggregator plugin does not support the _relative_ option supported by the Aggregator plugin, and always uses the _absolute_ sensor query mode.

650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
## Regressor Plugin <a name="regressorPlugin"></a>

The _Regressor_ plugin is able to perform regression of sensors, by using _random forest_ machine learning predictors. The algorithmic backend is provided by the OpenCV library.
For each input sensor in a certain unit, statistical features from recent values of its time series are computed. These are the average, standard deviation, sum of differences, 25th quantile and 75th quantile. These statistical features are then combined together in a single _feature vector_. 

In order to operate correctly, the models used by the regressor plugin need to be trained: this procedure is performed automatically online when using the _streaming_ mode, and can be triggered arbitrarily over the REST API. In _on demand_ mode, automatic training cannot be performed, and as such a pre-trained model must be loaded from a file.
The following are the configuration parameters available for the _Regressor_ plugin:

| Value | Explanation |
|:----- |:----------- |
| window | Length in milliseconds of the time window that is used to retrieve recent readings for the input sensors, starting from the latest one.
| trainingSamples | Number of samples necessary to perform training of the current model.
| targetDistance | Temporal distance (in terms of lags) of the sample that is to be predicted.
| inputPath | Path of a file from which a pre-trained random forest model must be loaded.
| outputPath | Path of a file to which the random forest model trained at runtime must be saved.
665
| getImportances | If true, the random forest will also compute feature importance values when trained, which are printed.
666
667
668

> NOTE &ensp;&ensp;&ensp;&ensp;&ensp; When the _duplicate_ option is enabled, the _outputPath_ field is ignored to avoid file collisions from multiple regressors.

669
670
> NOTE 2 &ensp;&ensp;&ensp;&ensp;&ensp; When loading the model from a file and getImportances is set to true, importance values will be printed only if the original model had this feature enabled upon training.

671
Additionally, input sensors in operators of the Regressor plugin accept the following parameter:
672
673
674

| Value | Explanation |
|:----- |:----------- |
675
| target | Boolean value. If true, this sensor represents the target for regression. Every unit in operators of the regressor plugin must have excatly one target sensor.
676
677
678
679
680
681

Finally, the Regressor plugin supports the following additional REST API action:

| Action | Explanation |
|:----- |:----------- |
| train | Triggers a new training phase for the random forest model. Feature vectors are temporarily collected in-memory until _trainingSamples_ vectors are obtained. Until this moment, the old random forest model is still used to perform prediction.
682
| importances | Returns the sorted importance values for the input features, together with the respective labels, if available.
683

684
685
686
687
688
689
690
691
692
693
694
695
## Tester Plugin <a name="testerPlugin"></a>
The _Tester_ plugin can be used to test the functionality and performance of the query engine, as well as of the unit system. It will perform a specified number of queries over the set of input sensors for each unit, and then output as a sensor the total number of retrieved readings. The following are the configuration parameters for operators in the _Tester_ plugin:

| Value | Explanation |
|:----- |:----------- |
| window | Length in milliseconds of the time window that is used to retrieve recent readings for the input sensors, starting from the latest one.
| queries | Number of queries to be performed at each computation interval. If more than the number of input sensors per unit, these will be looped over multiple times.
| relative | If true, the _relative_ query mode will be used. Otherwise the _absolute_ mode is used.

# Sink Plugins <a name="sinkplugins"></a>
Here we describe available plugins in DCDBAnalytics that are devoted to the output of sensor data (_sinks_), and that do not perform any analysis.

696
697
## File Sink Plugin <a name="filesinkPlugin"></a>
The _File Sink_ plugin allows to write the output of any other sensor to the local file system. As such, it does not produce output sensors by itself, and only reads from input sensors.
698
The input sensors can either be fully qualified, or can be described through the unit system. In this case, multiple input sensors can be generated automatically, and the respective output paths need to be adjusted by enabling the _autoName_ attribute described below, to prevent multiple sensors from being written to the same file. The file sink operators (named sinks) support the following attributes:
699
700
701
702
703
704
705
706
707
708
709

| Value | Explanation |
|:----- |:----------- |
| autoName | Boolean. If false, the output paths associated to sensors are interpreted literally, and a file is opened for them. If true, only the part in the path describing the current directory is used, while the file itself is named accordingly to the MQTT topic of the specific sensor.

Additionally, input sensors in sinks accept the following parameters:

| Value | Explanation |
|:----- |:----------- |
| path | The path to which the sensors's readings should be written. It is interpreted as described above for the _autoName_ attribute.

710
## Writing DCDB Analytics Plugins <a name="writingPlugins"></a>
711
712
Generating a DCDBAnalytics plugin requires implementing a _Operator_ and _Configurator_ class which contain all logic
tied to the specific plugin. Such classes should be derived from _OperatorTemplate_ and _OperatorConfiguratorTemplate_
713
respectively, which contain all plugin-agnostic configuration and runtime features. Please refer to the documentation 
Alessio Netti's avatar
Alessio Netti committed
714
of the _Aggregator_ plugin for an overview of how a basic plugin can be implemented.