Distributed ONOS

This section describes how to run ONOS in multi-instance mode.

Overview

One of ONOS's central goals is distributed functionality. An ONOS cluster is comprised of one or more ONOS instances, or nodes, running the same set of modules and sharing network state with each other. 

In this clustered mode, one can think of there being two types of control channel:

1. The Network control channel: This is between a switch and the ONOS instances. A switch may choose to establish a connection with one or more of the nodes in the ONOS cluster.
2. The Control network: This is between the nodes within the same cluster. 

To enable ONOS to work in multi-instance mode, one must load the bundles to enable clustering.

Checking for Clustering

We can check whether distributed functionality is enabled from the ONOS CLI by checking the output of list for the presence of onos-core-dist bundle:

onos> bundle:list | grep core-dist
154 | Active |  80 | 1.0.0.SNAPSHOT | onos-core-dist

On the other hand, if single-instance mode is desired, we would want to search for onos-core-trivial. 

Enabling Clustering

Clustering may be enabled statically or dynamically.

Statically (at startup)

There are two primary ways to specify which bundles for karaf to load at startup.

• Specify featuresBoot in org.apache.karaf.features.cfg.
  Karaf's startup behavior can be configured in this file. The featuresBoot list specifies the features that karaf should load when starting. We can add onos-core to this lineup so it looks like this:

  #
  # Comma separated list of features to install at startup
  #
  featuresBoot=config,standard,region,package,kar,ssh,management,webconsole,onos-api,onos-core,onos-cli,onos-openflow,onos-app-fwd,onos-app-mobility,onos-app-tvue

  We essentially replace onos-core-trivial with onos-core. Once edited and saved, ONOS should be restarted for it to take effect. 

  The downside to this method is that it must be done at each ONOS instance, if there are multiple running. 
  
  

• Customize how ONOS is packaged with ONOS_FEATURES.
  Alternatively, one can specify how the Karaf configuration file should be modified during the ONOS packaging process. The merit of this method is that the configuration will be applied automatically to any instance launched using the package. To do this, export the ONOS_FEATURES variable with the parameters for featuresBoot:

  $ export ONOS_FEATURES="webconsole,onos-api,onos-core,onos-cli,onos-openflow,onos-app-fwd,onos-app-mobility,onos-app-tvue"

  When onos-package is run again after the above is set, these parameters will be reflected in the Karaf configurations of any instances deployed with onos-install.

  For a way to persist, organize, and use multiple configurations without typing this out each time, refer to Test Cells.

Dynamically (at runtime)

Clustering can be enabled from the ONOS CLI by loading the distributed core. Note that one should unload the the single-instance core, if it is already running.

onos> feature:uninstall onos-core-trivial
onos> feature:install onos-core

This command may take a while as karaf searches and waits for dependencies to load. This must done at each node's CLI.

Switch Configuration

To enjoy scalability and high availability provided by ONOS, switches in the network should connect to all available controllers in the cluster.

In Mininet, this could be done switch by switch using ovs-vsctl set-controller.

mininet> sh ovs-vsctl set-controller <bridge> tcp:<ip1>:<port1> tcp:<ip2>:<port2> ...

It could be alternatively done once for all switches by script.

net = Mininet( controller=RemoteController )
c1 = net.addController( 'c1', ip='<ip1>', port=<port1> )
c2 = net.addController( 'c2', ip='<ip2>', port=<port2> )

Cluster Configuration and Management

Assign network interface for inter-node communication

A network interface should be assigned for ONOS nodes to communicate with each other.
This could simply be done through the environment variable ONOS_NIC. 
In the following example, we assume that controllers are running as VirtualBox VMs. In such case, the public network interface would be 192.168.56.0/24.

$ export ONOS_NIC=192.168.56.*

Next, we need to pack the configuration

$ onos-package

and deploy this configuration to each controller in the cluster.

$ onos-patch-vm <controller_ip>
$ onos-install -f <controller_ip>

By now, the node should be automatically connected with other nodes in the cluster.

Running multiple clusters 

ONOS uses Hazelcast for its cluster membership management. By default, ONOS clusters are configured to join the multicast group 224.2.2.3:54327. One may wish to change this if multiple, independent clusters need to coexist in the same domain. This may be done by modifying the values for multicast-group and multicast-port in hazelcast.xml:

            <multicast enabled="true">
                <multicast-group>224.1.2.3</multicast-group>
                <multicast-port>54327</multicast-port>
            </multicast>

Many cluster management command are available from the ONOS CLI, discussed in the remainder of this section.

Viewing members

The summary command gives a high-level overview of the cluster's status:

onos> summary
node=10.0.1.100, version=1.0.0.onosuser~2014/11/12@13:03
nodes=2, devices=14, links=63, hosts=0, clusters=1, paths=521, flows=0, intents=0

• node refers to the node whose CLI is being used to run the command (we'll call it the "management node" here)
• nodes shows the number of cluster members present

Note that clusters here refers to topology (i.e. the number of connected graphs), and has nothing to do with the status of the ONOS cluster.

roles displays a list of cluster members, with an asterisk next to the management node.

onos>  nodes
id=10.0.1.100, address=10.0.1.100:9876, state=ACTIVE *
id=10.0.1.200, address=10.0.1.200:9876, state=ACTIVE 

Mastership

Each device discovered by the cluster will have a primary master node responsible for controlling it, and zero or more standby nodes that can take the place of the master in the event that it fails. A node will take on the none role with respect to the device if it's not directly connected to it. This may happen if it learns about a device from another node, or if a formally connected device disconnects. 

The roles command lists the current roles of the nodes with respect to each device. Device mastership per node can be listed with masters. Currently, the master for each device is determined on a first-come basis, where the first node to discover the device becomes master, and subsequent nodes become standbys.

These assignments may be changed with the device-role command:

onos> masters
10.0.1.100: 1 devices
  of:5555000000000000
10.0.1.200: 1 devices
  of:1111000000000000
onos> device-role of:1111000000000000 10.0.1.200 standby
onos> masters
10.0.1.100: 2 devices
  of:1111000000000000
  of:5555000000000000
10.0.1.200: 0 devices

Note that while it is possible to assign mastership to an offline node, or set all nodes to standby for a device, a re-election will be triggered in the background to attempt to correct any misconfigurations (i.e. a single, functional node will be elected as master for the misconfigured device). 

Previous : The ONOS Web GUI
Next : Applications and Use Cases