Configuring Hybrid Messaging for TripleO

TripleO Messaging

The OpenStack oslo.messaging library provides RPC and Notification messaging communication patterns for control plane services. The RPC interface is used for interactive invocation and control of services while the Notification interface provides a pub-sub pattern that can be used for event generation and updates.

During the tripleo rocky release, the configuration of the oslo.messaging services has been updated. The update enables the separation of the messaging backends used for RPC and Notifications and also enables the use of different messaging backend server types (e.g. in addition to rabbitmq). The deployment configurations that can be defined include the following:

  • The standard deployment of a shared rabbitmq server messaging backend (e.g. cluster) that is used for both RPC and Notification communications
  • A hybrid deployment of an apache dispatch router messaging backend (qdrouterd) for RPC communications (e.g. via the oslo.messaging AMQP 1.0 driver) and a rabbitmq server messaging backend for Notification communications

Oslo.Messaging Services

Since the mitaka release, oslo.messaging has supported the deployment of separate messaging backends for RPC and Notification communications. To support separate messaging backends in tripleo, oslo.messaging services for RPC and Notify were introduced in place of the stand alone rabbitmq service. This layering allows for the RPC and Notification messaging services to be separately controlled and configured.

Standard Deployment of Single RabbitMQ Server Backend

The standard deployment of a single rabbitmq server backend (e.g. cluster) is retained by mapping both oslo.messaging RPC and Notify services to the same messaging transport.

OS::TripleO::Services::OsloMessagingRpc:docker/services/messaging/rpc-rabbitmq.yaml
OS::TripleO::Services::OsloMessagingNotify:docker/services/messaging/notify-rabbitmq-shared.yaml

An examination of these service reveals that rpc-rabbitmq.yaml instantiates the rabbitmq server role while notify-rabbitmq-shared.yaml maps the oslo.messaging global configuration setting for Notify to the same rabbitmq server backend that is used for RPC communications.

During the deployment, the parameters used for the configuration of the oslo.messaging RPC and Notification services are used to generate the messaging transport configuration defined for each service. An example of this can be seen in the puppet-tripleo base profile for nova.

class { '::nova':
  default_transport_url      => os_transport_url({
    'transport' => $oslomsg_rpc_proto,
    'hosts'     => $oslomsg_rpc_hosts,
    'port'      => $oslomsg_rpc_port,
    'username'  => $oslomsg_rpc_username,
    'password'  => $oslomsg_rpc_password,
    'ssl'       => $oslomsg_rpc_use_ssl_real,
  }),
  notification_transport_url => os_transport_url({
    'transport' => $oslomsg_notify_proto,
    'hosts'     => $oslomsg_notify_hosts,
    'port'      => $oslomsg_notify_port,
    'username'  => $oslomsg_notify_username,
    'password'  => $oslomsg_notify_password,
    'ssl'       => $oslomsg_notify_use_ssl_real,
  }),
}

Following deployment, an example configuration of the oslo.messaging services can be checked in the /etc/nova/nova.conf configuration file. In it, you will see that the same transport is used for RPC and Notifications to the shared rabbitmq server backend.

[DEFAULT]
transport_url=rabbit://guest:secrete@host1.internalapi.localdomain:5672/?ssl=0
.
.
. 
[oslo_messaging_notifications]
transport_url=rabbit://guest:secrete@host1.internalapi.localdomain:5672/?ssl=0Hybrid Messaging Deployment

The deployment of two independent messaging backends is realized by mapping the oslo.messaging RPC and Notification transports to different messaging systems. For this deployment, the apache dispatch router (qdrouterd) is used as the RPC messaging backend in place of the rabbitmq server. The qdrouterd server is integrated with the oslo.messaging AMQP 1.0 driver and provides direct messaging capabilities for the RPC communications. The rabbitmq server is used as the messaging backend for the Notification communications where the broker storage functionality is necessary.

This hybrid messaging example environment in tripleo enables the dual messaging backend deployment for RPC and Notifications:

# *******************************************************************
# This file was created automatically by the sample environment
# generator. Developers should use `tox -e genconfig` to update it.
# Users are recommended to make changes to a copy of the file instead
# of the original, if any customizations are needed.
# *******************************************************************
# title: Hybrid qdrouterd for rpc and rabbitmq for notify messaging backend
# description: |
#   Include this environment to enable hybrid messaging backends for
#   oslo.messaging rpc and notification services
parameter_defaults:
  # The network port for messaging Notify backend
  # Type: number
  NotifyPort: 5672

  # The network port for messaging backend
  # Type: number
  RpcPort: 31459

resource_registry:
  OS::TripleO::Services::OsloMessagingNotify: ../../docker/services/messaging/notify-rabbitmq.yaml
  OS::TripleO::Services::OsloMessagingRpc: ../../docker/services/messaging/rpc-qdrouterd.yaml

An examination of these services reveals that the rpc-qdrouterd.yaml instantiates the qdrouterd server role to provide RPC communications via direct messaging while notify-rabbitmq.yaml instantiates the rabbitmq server role to provide Notify communications via the broker’s queues.

Following deployment, the resulting configuration of the oslo.messaging services can be checked in the /etc/nova/nova.conf configuration file. For the deployment, the RPC transport is defined to use the AMQP 1.0 driver with the qdrouterd messaging backend and the Notification transport is defined to use the rabbit driver with the corresponding rabbitmq messaging backend.

[DEFAULT]
transport_url=amqp://guest:secrete@cphost.internalapi.localdomain:31459/?ssl=0
.
.
.
[oslo_messaging_notifications]
transport_url=rabbit://guest:secrete@cphost.internalapi.localdomain:5672/?ssl=0

Add the following arguments to your openstack overcloud deploy command to deploy with separate messaging backends:

openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/messaging/rpc-qdrouterd-notify-rabbitmq-hybrid.yaml

Summary

The configuration of oslo.messaging services has been updated in the rocky release. These updates simplify the deployment of the messaging backend systems so that operators can begin to evaluate and understand hybrid messaging and evaluate performance and scalability benefits.

Advertisements