This section describes what typically comprises an ONOS test environment, and how to manage multiple ONOS instances using cells and utility scripts.

Overview

Testing and managing a distributed platform like ONOS can become a cumbersome task. 

In order to simplify testing and to make it more repeatable, a number of assets, including the aforementioned ONOS scripts, have been developed to make the developer’s  and tester’s lives easier. These are located under onos/tools/test/, and Appendix A of this Guide provides a listing of available utilities.

Test Environment Components

An ONOS developer's environment may include the following:

• Development/build machine : for actual code development e.g. running the IDE and building/packaging/deploying ONOS. 
• One or more test deployment VMs : for running ONOS instances
• A Mininet VM : for emulating networks to test with ONOS

In the above case, the ONOS scripts are run from the development machine, and are directed towards the VMs running on a hypervisor (which can also be on the same machine, or elsewhere). 

Lastly, since a developer may want to test different scenarios against different sets of VMs or servers, ONOS provides the notion of test cells. 

Test Cells

A test cell refers to a specific controller cluster environment designated for testing. This can mean a set of bare-metal servers, or a set of VMs running on developer’s laptop. Cell definition files are bash snippets that export a few environment variables into the developer’s (or tester’s) shell. These variables are then used by a number of the ONOS test and utility scripts. This allows the scripts and test scenarios to be independent to a specific test bench setup.

Using cells

Cell definitions are loaded into the shell environment with the cell utility. This utility takes the name of a cell definition file as the argument. For example:

$ cell local                                              
ONOS_CELL=local
OCI=192.168.56.101
OC1=192.168.56.101
OC2=192.168.56.102
OCN=192.168.56.103
ONOS_FEATURES=webconsole,onos-api,onos-core,onos-cli,onos-openflow,onos-gui,onos-rest,onos-app-fwd,onos-app-proxyarp,onos-app-tvue
ONOS_NIC=192.168.56.*

cell echoes back with the key environment variables understood by the ONOS scripts:

• OCI : the default target node IP. This is an alias for OC1. 
• OC[1-3] : IP addresses of the VMs hosting ONOS instances. More OC instances may be set, if necessary.
• OCN : IP address of the VM with Mininet
• ONOS_FEATURES : a comma-separated list of bundle names, loaded at startup by an ONOS instance within this cell
• ONOS_NIC : The address block used amongst ONOS instances for inter-controller (clustering) and OpenFlow communication 

Once a cell definition is exported, utilities such as onos will fall back to using the value set in OCI (192.168.56.101 for above) if not given any parameters. 

The utility script cells can be used to view all available cell definitions. Without parameters, cell will list the values associated with the current cell in use.

Cell Definition Files

A cell definition file defines values for the aforementioned environment variables. A cell definition file may look like the following:

# Default ONOS Controller VM instances 1,2,3 and a Mininet VM
export OC1="192.168.56.101"
export OC2="192.168.56.102"
export OC3="192.168.56.104"
export OCN="192.168.56.103"  # Mininet VM
       
# for node clustering
export ONOS_NIC=”192.168.56.*”    

# ONOS features to load
export ONOS_FEATURES="webconsole,onos-api,onos-core-trivial,onos-cli"

These files are located in ONOS_ROOT/tools/test/cells/.

ONOS 1.1.0 and later comes with vicell command to ease editing and applying cell definition files. See vicell -h for usage details.

Creating custom cell definitions

Custom cell definition files may be added to ONOS_ROOT/tools/test/cells/, which enables them to be loaded and listed with cell and cells.

Test VM Setup

ONOS, being an SDN controller authored in Java, can run on a variety of platforms. However, in the interest of focus, the ONOS team engages primarily in testing on Ubuntu server distributions, specifically Ubuntu Server 14.04* LTS 64-bit.

Additionally, the VMs used for running ONOS instances and Mininet can be configured to better mesh with the functionality of the ONOS utility scripts and the test environment:

• Adding two network interfaces : One should be configured to use the host adapter and the other to use either the NAT or Bridged adapter to allow outside access.
• Adding an user named 'sdn' : The scripts check for the environment variable ONOS_USER, and, if it is unset, uses sdn as the default username. 
• Allowing password-less (key-based) login : The scripts rely on ssh and scp for their functions, and this lets one avoid having to type a password each time a script is run.

For password-less login, the onos-push-keys  utility can be used to transfer one's SSH key to the VM.

Using the Test Environment

Once set up, changes to the codebase can be (relatively) quickly tested in a distributed setting as follows:

1. Make changes to the code
2. Build with mvn clean install
3. Load the cell settings with cell <your_cell_name>
   
4. Package executables for deployment with onos-package 
5. Deploy to test VMs in the cell

For step 5, a developer can take advantage of the OC variables when using onos-install:

$ onos-install -f $OC1  #install ONOS to OC1

This procedure must be repeated for each target, as onos-install is only capable of handling one target at a time. In ONOS 1.1.0 and later, the onos-group command can be used to automatically target all cell member VMs at once:

$ onos-group install -f

Utilities such as onos-service also take the --cell argument that enables it to manage all cell members at once. For example, to restart all ONOS instances in a cell:

$ onos-service --cell restart

Using Lightweight Virtualization

Finally, a developer may not be able to run multiple VMs due to the resource constraints of their build machine or environment. In such cases, lightweight virtualization technologies may be used to supply the resource containment for each ONOS instance. Multiple instances using LXC demonstrates the use of LXC containers for this purpose.

Previous : Development Environment Setup
Next : Continuous Integration