Data collection¶

Middleware for the OpenStack Object Storage service¶

A subset of Object Store statistics requires additional middleware to be installed behind the proxy of Object Store. This additional component emits notifications containing data-flow-oriented meters, namely the storage.objects.(incoming|outgoing).bytes values. The list of these meters are listed in OpenStack Object Storage, marked with notification as origin.

The instructions on how to install this middleware can be found in Configure the Object Storage service for Telemetry section in the OpenStack Installation Guide.

Telemetry middleware¶

Telemetry provides the capability of counting the HTTP requests and responses for each API endpoint in OpenStack. This is achieved by storing a sample for each event marked as audit.http.request, audit.http.response, http.request or http.response.

It is recommended that these notifications be consumed as events rather than samples to better index the appropriate values and avoid massive load on the Metering database. If preferred, Telemetry can consume these events as samples if the services are configured to emit http.* notifications.

Central agent¶

As the name of this agent shows, it is a central component in the Telemetry architecture. This agent is responsible for polling public REST APIs to retrieve additional information on OpenStack resources not already surfaced via notifications, and also for polling hardware resources over SNMP.

The following services can be polled with this agent:

• OpenStack Networking
• OpenStack Object Storage
• OpenStack Block Storage
• Hardware resources via SNMP
• Energy consumption meters via Kwapi framework

To install and configure this service use the Install the Telemetry module section in the OpenStack Installation Guide.

The central agent does not need direct database connection. The samples collected by this agent are sent via AMQP to the collector service or any external service, which is responsible for persisting the data into the configured database back end.

Compute agent¶

This agent is responsible for collecting resource usage data of VM instances on individual compute nodes within an OpenStack deployment. This mechanism requires a closer interaction with the hypervisor, therefore a separate agent type fulfills the collection of the related meters, which is placed on the host machines to locally retrieve this information.

A compute agent instance has to be installed on each and every compute node, installation instructions can be found in the Install the Compute agent for Telemetry section in the OpenStack Installation Guide.

Just like the central agent, this component also does not need a direct database connection. The samples are sent via AMQP to the collector.

The list of supported hypervisors can be found in Supported hypervisors. The compute agent uses the API of the hypervisor installed on the compute hosts. Therefore the supported meters may be different in case of each virtualization back end, as each inspection tool provides a different set of meters.

The list of collected meters can be found in OpenStack Compute. The support column provides the information that which meter is available for each hypervisor supported by the Telemetry module.

Support for HA deployment of the central and compute agent services¶

Both the central and the compute agent can run in an HA deployment, which means that multiple instances of these services can run in parallel with workload partitioning among these running instances.

The Tooz library provides the coordination within the groups of service instances. It provides an API above several back ends that can be used for building distributed applications.

Tooz supports various drivers including the following back end solutions:

• Zookeeper. Recommended solution by the Tooz project.
• Redis. Recommended solution by the Tooz project.
• Memcached. Recommended for testing.

You must configure a supported Tooz driver for the HA deployment of the Telemetry services.

For information about the required configuration options that have to be set in the ceilometer.conf configuration file for both the central and compute agents, see the Coordination section in the OpenStack Configuration Reference.

The availability check of the instances is provided by heartbeat messages. When the connection with an instance is lost, the workload will be reassigned within the remained instances in the next polling cycle.

For backward compatibility and supporting existing deployments, the central agent configuration also supports using different configuration files for groups of service instances of this type that are running in parallel. For enabling this configuration set a value for the partitioning_group_prefix option in the Central section in the OpenStack Configuration Reference.

To enable the compute agent to run multiple instances simultaneously with workload partitioning, the workload_partitioning option has to be set to True under the Compute section in the ceilometer.conf configuration file.

IPMI agent¶

This agent is responsible for collecting IPMI sensor data and Intel Node Manager data on individual compute nodes within an OpenStack deployment. This agent requires an IPMI capable node with the ipmitool utility installed, which is commonly used for IPMI control on various Linux distributions.

An IPMI agent instance could be installed on each and every compute node with IPMI support, except when the node is managed by the Bare metal service and the conductor.send_sensor_data option is set to true in the Bare metal service. It is no harm to install this agent on a compute node without IPMI or Intel Node Manager support, as the agent checks for the hardware and if none is available, returns empty data. It is suggested that you install the IPMI agent only on an IPMI capable node for performance reasons.

Just like the central agent, this component also does not need direct database access. The samples are sent via AMQP to the collector.

The list of collected meters can be found in Bare metal service.

Pipeline configuration¶

Pipeline configuration by default, is stored in separate configuration files, called pipeline.yaml and event_pipeline.yaml, next to the ceilometer.conf file. The meter pipeline and event pipeline configuration files can be set by the pipeline_cfg_file and event_pipeline_cfg_file options listed in the Description of configuration options for api table section in the OpenStack Configuration Reference respectively. Multiple pipelines can be defined in one pipeline configuration file.

The meter pipeline definition looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

---
sources:
  - name: 'source name'
    interval: 'how often should the samples be injected into the pipeline'
    meters:
      - 'meter filter'
    resources:
      - 'list of resource URLs'
    sinks
      - 'sink name'
sinks:
  - name: 'sink name'
    transformers: 'definition of transformers'
    publishers:
      - 'list of publishers'

The interval parameter in the sources section should be defined in seconds. It determines the polling cadence of sample injection into the pipeline, where samples are produced under the direct control of an agent.

There are several ways to define the list of meters for a pipeline source. The list of valid meters can be found in Measurements. There is a possibility to define all the meters, or just included or excluded meters, with which a source should operate:

• To include all meters, use the * wildcard symbol. It is highly advisable to select only the meters that you intend on using to avoid flooding the metering database with unused data.
• To define the list of meters, use either of the following:
  • To define the list of included meters, use the meter_name syntax.
  • To define the list of excluded meters, use the !meter_name syntax.
  • For meters, which have variants identified by a complex name field, use the wildcard symbol to select all, e.g. for “instance:m1.tiny”, use “instance:*”.

The above definition methods can be used in the following combinations:

• Use only the wildcard symbol.
• Use the list of included meters.
• Use the list of excluded meters.
• Use wildcard symbol with the list of excluded meters.

The optional resources section of a pipeline source allows a static list of resource URLs to be configured for polling.

The transformers section of a pipeline sink provides the possibility to add a list of transformer definitions. The available transformers are:

The publishers section contains the list of publishers, where the samples data should be sent after the possible transformations.

Similarly, the event pipeline definition looks like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

---
sources:
  - name: 'source name'
    events:
      - 'event filter'
    sinks
      - 'sink name'
sinks:
  - name: 'sink name'
    publishers:
      - 'list of publishers'

The event filter uses the same filtering logic as the meter pipeline.

1
2
3
4
5
6
7
8

transformers:
    - name: "rate_of_change"
      parameters:
          target:
              name: "cpu_util"
              unit: "%"
              type: "gauge"
              scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

transformers:
    - name: "rate_of_change"
      parameters:
          source:
              map_from:
                  name: "disk\\.(read|write)\\.(bytes|requests)"
                  unit: "(B|request)"
          target:
              map_to:
                  name: "disk.\\1.\\2.rate"
                  unit: "\\1/s"
              type: "gauge"

1
2
3
4
5
6
7

transformers:
    - name: "unit_conversion"
      parameters:
          target:
              name: "disk.kilobytes"
              unit: "KB"
              scale: "1.0 / 1024.0"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

transformers:
    - name: "unit_conversion"
      parameters:
          source:
              map_from:
                  name: "disk\\.(read|write)\\.bytes"
          target:
              map_to:
                  name: "disk.\\1.kilobytes"
              scale: "1.0 / 1024.0"
              unit: "KB"

1
2
3
4
5

transformers:
    - name: "aggregator"
      parameters:
          retention_time: 60
          resource_metadata: last

1
2
3
4
5
6

transformers:
    - name: "aggregator"
      parameters:
          size: 15
          user_id: first
          resource_metadata: drop

transformers:
    - name: "accumulator"
      parameters:
          size: 15

memory_util = 100 * memory.usage / memory

1
2
3
4
5
6
7
8

transformers:
    - name: "arithmetic"
      parameters:
        target:
          name: "memory_util"
          unit: "%"
          type: "gauge"
          expr: "100 * $(memory.usage) / $(memory)"

1
2
3
4
5
6
7
8

transformers:
    - name: "arithmetic"
      parameters:
        target:
          name: "avg_cpu_per_core"
          unit: "ns"
          type: "cumulative"
          expr: "$(cpu) / ($(cpu).resource_metadata.cpu_number or 1)"

Meter definitions¶

The Telemetry module collects a subset of the meters by filtering notifications emitted by other OpenStack services. Starting with the Liberty release, you can find the meter definitions in a separate configuration file, called ceilometer/meter/data/meter.yaml. This enables operators/administrators to add new meters to Telemetry project by updating the meter.yaml file without any need for additional code changes.

A standard meter definition looks like the following:

1
2
3
4
5
6
7
8
9

---
metric:
  - name: 'meter name'
    event_type: 'event name'
    type: 'type of meter eg: gauge, cumulative or delta'
    unit: 'name of unit eg: MB'
    volume: 'path to a measurable value eg: $.payload.size'
    resource_id: 'path to resouce id eg: $.payload.id'
    project_id: 'path to project id eg: $.payload.owner'

The definition above shows a simple meter definition with some fields, from which name, event_type, type, unit, and volume are required. If there is a match on the event type, samples are generated for the meter.

If you take a look at the meter.yaml file, it contains the sample definitions for all the meters that Telemetry is collecting from notifications. The value of each field is specified by using json path in order to find the right value from the notification message. In order to be able to specify the right field you need to be aware of the format of the consumed notification. The values that need to be searched in the notification message are set with a json path starting with $. For instance, if you need the size information from the payload you can define it like $.payload.size.

A notification message may contain multiple meters. You can use * in the meter definition to capture all the meters and generate samples respectively. You can use wild cards as shown in the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

---
metric:
  - name: $.payload.measurements.[*].metric.[*].name
    event_type: 'event_name.*'
    type: 'delta'
    unit: $.payload.measurements.[*].metric.[*].unit
    volume: payload.measurements.[*].result
    resource_id: $.payload.target
    user_id: $.payload.initiator.id
    project_id: $.payload.initiator.project_id

In the above example, the name field is a json path with matching a list of meter names defined in the notification message.

You can even use complex operations on json paths. In the following example, volume and resource_id fields perform an arithmetic and string concatenation:

1
2
3
4
5
6
7
8

---
metric:
- name: 'compute.node.cpu.idle.percent'
  event_type: 'compute.metrics.update'
  type: 'gauge'
  unit: 'percent'
  volume: payload.metrics[?(@.name='cpu.idle.percent')].value * 100
  resource_id: $.payload.host + "_" + $.payload.nodename

You will find some existence meters in the meter.yaml. These meters have a volume as 1 and are at the bottom of the yaml file with a note suggesting that these will be removed in Mitaka release.

For example, the meter definition for existence meters is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

---
metric:
  - name: 'meter name'
    type: 'delta'
    unit: 'volume'
    volume: 1
    event_type:
        - 'event type'
    resource_id: $.payload.volume_id
    user_id: $.payload.user_id
    project_id: $.payload.tenant_id

These meters are not loaded by default. To load these meters, flip the disable_non_metric_meters option in the ceilometer.conf file.

Database dispatcher¶

When the database dispatcher is configured as data store, you have the option to set a time_to_live option (ttl) for samples. By default the time to live value for samples is set to -1, which means that they are kept in the database forever.

The time to live value is specified in seconds. Each sample has a time stamp, and the ttl value indicates that a sample will be deleted from the database when the number of seconds has elapsed since that sample reading was stamped. For example, if the time to live is set to 600, all samples older than 600 seconds will be purged from the database.

Certain databases support native TTL expiration. In cases where this is not possible, a command-line script, which you can use for this purpose is ceilometer-expirer. You can run it in a cron job, which helps to keep your database in a consistent state.

The level of support differs in case of the configured back end:

HTTP dispatcher¶

The Telemetry module supports sending samples to an external HTTP target. The samples are sent without any modification. To set this option as the collector’s target, the dispatcher has to be changed to http in the ceilometer.conf configuration file. For the list of options that you need to set, see the see the dispatcher_http section in the OpenStack Configuration Reference.

File dispatcher¶

You can store samples in a file by setting the dispatcher option in the ceilometer.conf file. For the list of configuration options, see the dispatcher_file section in the OpenStack Configuration Reference.

Gnocchi dispatcher¶

The Telemetry module supports sending the metering data to Gnocchi back end through the gnocchi dispatcher. To set this option as the target, change the dispatcher to gnocchi in the ceilometer.conf configuration file.

For the list of options that you need to set, see the dispatcher_gnocchi section in the OpenStack Configuration Reference.