Skip to content
GitLab
Menu
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
dcdb
dcdb
Commits
b1f6f664
Commit
b1f6f664
authored
Jun 13, 2019
by
Alessio Netti
Browse files
Analytics: update Readme
parent
37a44e0d
Changes
1
Show whitespace changes
Inline
Side-by-side
analytics/README.md
View file @
b1f6f664
...
...
@@ -12,12 +12,15 @@
1.
[
Configuration Syntax
](
#configSyntax
)
2.
[
Instantiating Units
](
#instantiatingUnits
)
3.
[
MQTT Topics
](
#mqttTopics
)
4.
[
Pipelining Analyzers
](
#pipelining
)
5.
[
Job Analyzers
](
#jobanalyzers
)
3.
[
Rest API
](
#restApi
)
1.
[
List of ressources
](
#listOfRessources
)
2.
[
Examples
](
#restExamples
)
3.
[
Plugins
](
#plugins
)
1.
[
Average Plugin
](
#averagePlugin
)
2.
[
Writing Plugins
](
#writingPlugins
)
1.
[
Aggregator Plugin
](
#averagePlugin
)
2.
[
Job Aggregator Plugin
](
#jobaveragePlugin
)
3.
[
Writing Plugins
](
#writingPlugins
)
# Introduction <a name="introduction"></a>
In this Readme we describe the DCDB Data Analytics framework, and all data abstractions that are associated with it.
...
...
@@ -234,15 +237,15 @@ mqttPart /avg1
}
output {
sensor "<bottomup, filter cpu0
1
>sum" {
sensor "<bottomup, filter cpu0
0
>sum" {
mqttsuffix /sum
}
sensor "<bottomup, filter cpu0
1
>max" {
sensor "<bottomup, filter cpu0
0
>max" {
mqttsuffix /max
}
sensor "<bottomup, filter cpu0
1
>avg" {
sensor "<bottomup, filter cpu0
0
>avg" {
mqttsuffix /avg
}
}
...
...
@@ -306,6 +309,9 @@ rack02.node03.cpu00. {
}
```
> NOTE       In order to get units that operate at the _node_ level, the output sensors in the
configuration discussed here should have a unit block in the form of
<
bottomup
1,
filter
cpu00
>
.
#### MQTT Topics <a name="mqttTopics"></a>
The MQTT topics associated to output sensors of a certain analyzer are constructed in different ways depending
on the unit they belong to:
...
...
@@ -326,6 +332,27 @@ the framework cannot infer the correct order of initialization so as to resolve
> NOTE       This feature is not supported when using analyzers in _on demand_ mode.
#### Job Analyzers <a name="jobanalyzers"></a>
_Job Analyzers_
are a class of analyzers which act on job-specific data. Such data is structured in _job units_. These
work similarly to ordinary units, with the difference that they are arranged hierarchically as follows:
*
The top unit is associated to the job itself and contains all of the required output sensors;
*
One sub-unit for each node on which the job was running is allocated. Each of these sub-units contains all of the input
sensors that are required at configuration time.
The computation algorithms driving job analyzers can then navigate freely this hierarchical unit design according to
their specific needs. Since all of the sub-units are tied to the nodes on which the job was running, the output sensors
specified in the configuration must also be at the node level in the unit system, such that unit resolution is performed correctly.
Job analyzers also support the _streaming_ and _on demand_ modes, which work like the following:
*
In
**streaming**
mode, the job analyzer will retrieve the list of jobs that were running in the time interval starting
from the last computation to the present; it will then build one job unit for each of them, and subsequently perform computation;
*
In
**on demand**
mode, users can query a specific job id, for which a job unit is built and computation is performed.
> NOTE       The _duplicate_ setting does not affect job analyzers.
## Rest API <a name="restApi"></a>
DCDBAnalytics provides a REST API that can be used to perform various management operations on the framework. The
API is functionally identical to that of DCDBPusher, and is hosted at the same address. All requests that are targeted
...
...
@@ -601,6 +628,12 @@ Additionally, output sensors in analyzers of the Aggregator plugin accept the fo
| operation | Operation to be performed over the input sensors. Can be "sum", "average", "maximum", "minimum", "std" or "percentiles".
| percentile | Specific percentile to be computed when using the "percentiles" operation. Can be an integer in the (0,100) range.
## Job Aggregator Plugin <a name="jobaveragePlugin"></a>
The _Job Aggregator_ plugin offers the same functionality as the _Aggregator_ plugin, but on a per-job basis. As such,
it performs aggregation of the specified input sensors across all nodes on which each job is running. Please refer
to the corresponding
[
section
](
#jobanalyzers
)
for more details.
## Writing DCDBAnalytics Plugins <a name="writingPlugins"></a>
Generating a DCDBAnalytics plugin requires implementing a _Analyzer_ and _Configurator_ class which contain all logic
tied to the specific plugin. Such classes should be derived from _AnalyzerTemplate_ and _AnalyzerConfiguratorTemplate_
...
...
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment