The workflow is recommended for developers who wish to use cells and the stc / onos-* utility scripts to manage an ONOS development or testing cluster using multiple VMs or hardware servers.

Overview

Testing and managing a distributed platform like ONOS can become a cumbersome task. 

In order to simplify testing and to make it more repeatable, a number of assets, including the aforementioned ONOS scripts, have been developed to make the developer’s  and tester’s lives easier. These are located under onos/tools/test/, and Appendix A of this Guide provides a listing of available utilities. From your management (in this case development) machine, with few commands you'll be able to:

• Deploy ONOS as a cluster on one or multiple remote target machines
• Deploy an arbitrary Mininet topology on a remote machine and let its virtual switches connect to the ONOS instances mentioned above
• Test that the ONOS cluster and the Mininet topology came up correctly (i.e. check automatically ONOS logs, number of switches, links, hosts, ...)

Test Environment Components

Following the model reported in the Abstractions and Definitions section, an ONOS developer's environment may include the following:

• Management/Development/build machine: for actual code development e.g. running the IDE and building/packaging/deploying ONOS. 
• One or more ONOS target machines: for running ONOS instances
• A Mininet target machine: for emulating networks to test with ONOS

In the above case, the ONOS scripts are run from the management machine, and are directed towards the target machines. The target machines (both ONOS and Mininet target machines) can be either local or remote, physical, virtual, ...

Target machines requirements

The following requirements should be satisfied in order to be able to push software configurations on the remote target machines.

All target machines:

• Be all-to-all reachable at IP layer (basically all machines -including the management machine- need to be able to ping one each other)
• Have a user sdn that can be sudoer on the machine, without any need to insert the password for it
• Accessing without password from the management machine to the target machines. To do this the public key of the user on the management machine must be present in the .ssh/authorized_key file on all your target machines.
• So essentially as a proof that everything works, from your management machine you should be able to login into any target machine doing ssh sdn@TARGET_MACHINE_IP and execute sudo echo hello (for example) with no passwords required

ONOS target machines

Additionally, ONOS target machines should have

• Oracle Java (at least JRE) 1.8 installed

Mininet target machine

Additionally, the Mininet target machine should have

• Mininet install. This can be simple as typing sudo apt-get install mininet. Anyway, for more, and more detailed instructions, please refer directly to the Mininet website

Test Cells

Since a developer may want to test different scenarios against different sets of VMs or servers, ONOS provides the notion of test cells.

A test cell refers to a specific target machines environment designated for testing. Cell definition files are bash snippets that simply export few environment variables into the developer’s (or tester’s) shell. These variables are then used by a number of the ONOS test and utility scripts. This allows the scripts and test scenarios to be independent to a specific test bench setup.

Using cells

Cell definitions are loaded into the shell environment with the cell utility. This utility takes the name of a cell definition file as the argument. For example:

$ cell local                                              
ONOS_CELL=local
OCI=192.168.56.101
OC1=192.168.56.101
OC2=192.168.56.102
OCN=192.168.56.103
ONOS_FEATURES=openflow
ONOS_NIC=192.168.56.*

Cell files attributes

Below it's reported a list of the basic attributes you can express and find in a cell file.

• OCI : the default target machine node IP. This is an alias for OC1. 
• OC[1-N] : the IP addresses of the ONOS target machines. You can set as many OC instances as you want (depending on the number of ONOS target machines you want to run)
• OCN : the IP address of the Mininet target machine
• ONOS_FEATURES : a comma-separated list of bundle names, loaded at startup by an ONOS instances within this cell
• ONOS_NIC : The address block used amongst ONOS instances for inter-controller (clustering) and OpenFlow communication. In the example above, it's assumed that all ONOS instances have an IP address in the subnet 192.168.56.0/24

Once a cell definition is exported, utilities such as onos will fall back to using the value set in OCI (192.168.56.101 for above) if not given any parameters. 

The utility script cells can be used to view all available cell definitions. Without parameters, cell will list the values associated with the current cell in use.

Defining cell files

Cells files can be found on your management machine, under $ONOS_ROOT/tools/test/cells, where $ONOS_ROOT is the onos folder containing the ONOS repository files cloned.

When a file is added to this directory, it enables it to be loaded and listed with the cell and cells commands.

A cell definition file defines values for the aforementioned environment variables. A cell definition file may look like the following:

# Defining a cell composed by 3 ONOS target machines and one Mininet target machine

export OCI="192.168.56.101"     # Default ONOS target machine (ONOS 1)
export OC1="192.168.56.101"     # ONOS target machine 1
export OC2="192.168.56.102"     # ONOS target machine 2
export OC3="192.168.56.103"     # ONOS target machine 1
export OCN="192.168.56.110"     # Mininet target machine
export ONOS_NIC=”192.168.56.*”    

# ONOS features to load
export ONOS_FEATURES="openflow"

Test VM Setup

ONOS, being an SDN controller authored in Java, can run on a variety of platforms. However, in the interest of focus, the ONOS team engages primarily in testing on Ubuntu server distributions, specifically Ubuntu Server 14.04* LTS 64-bit.

Additionally, the VMs used for running ONOS instances and Mininet can be configured to better mesh with the functionality of the ONOS utility scripts and the test environment:

• Adding two network interfaces : One should be configured to use the host adapter and the other to use either the NAT or Bridged adapter to allow outside access.
• Adding an user named 'sdn' : The scripts check for the environment variable ONOS_USER, and, if it is unset, uses sdn as the default username. 
• Allowing password-less (key-based) login : The scripts rely on ssh and scp for their functions, and this lets one avoid having to type a password each time a script is run.

For password-less login, the onos-push-keys  utility can be used to transfer one's SSH key to the VM.

Using the Test Environment

Once set up, changes to the codebase can be (relatively) quickly tested in a distributed setting as follows:

1. Make changes to the code
2. Build with mvn clean install
3. Load the cell settings with cell <your_cell_name>
   
4. Package executables for deployment with onos-package 
5. Deploy to test VMs in the cell

For step 5, a developer can take advantage of the OC variables when using onos-install:

$ onos-install -f $OC1  #install ONOS to OC1

This procedure must be repeated for each target, as onos-install is only capable of handling one target at a time. In ONOS 1.1.0 and later, the onos-group command can be used to automatically target all cell member VMs at once:

$ onos-group install -f

Utilities such as onos-service also take the --cell argument that enables it to manage all cell members at once. For example, to restart all ONOS instances in a cell:

$ onos-service --cell restart

Alternative: Lightweight Virtualization

A developer might not wish to run multiple VMs due to resource constraints of the build machine or environment. In such cases, lightweight virtualization technologies may be used to supply the resource containment for each ONOS instance. One easy way to create an ONOS cluster in a single VM or server is to use onos.py, which uses Mininet hosts/containers for ONOS nodes. Another approach is to create multiple containers using lxc.