This section describes what typically comprises an ONOS test environment, and how to manage multiple ONOS instances using cells and utility scriptsThe workflow is recommended for developers who wish to use cells and the stc / onos-* utility scripts to manage an ONOS development or testing cluster using multiple VMs or hardware servers.

Overview

Testing and managing a distributed platform like ONOS can become a cumbersome task. 

In order to simplify testing and to make it more repeatable, a number of assets, including the aforementioned ONOS scripts, have been developed to make the developer’s  and tester’s lives easier. These are located under onos $ONOS_ROOT/tools/test/, and . Appendix A of this Guide guide provides a listing of available utilities. From your management (in this case development) machine, with few commands you'll be able to:

• Deploy ONOS as a cluster on one or multiple remote target machines
• Deploy an arbitrary Mininet topology on a remote machine and let its the switches connect to the ONOS machines mentioned above
• Test that the ONOS cluster and the Mininet topology came up correctly (i.e. check automatically ONOS logs, number of switches, links, hosts, ...)

Test Environment Components

An Following the model reported in the Abstractions and Definitions section, an ONOS developer's environment may include the following:

• Management/Development/build machine: for actual code development - i.e. g. running the IDE and building/packaging/deploying ONOS. 
• One or more test deployment VMs ONOS target machines: for running ONOS instances
• A Mininet VM target machine: for emulating networks to test with ONOS

In the above case, the ONOS scripts are run from the development management machine, and are directed towards the VMs running on a hypervisor (which can also be on the same machine, or elsewhere). target machines. The target machines (both ONOS and Mininet target machines) can be either local or remote, physical, virtual, ...

Target machines requirements

The following requirements should be satisfied in order to be able to push software configurations on the remote target machines.

All target machines:

• Be all-to-all reachable at IP layer (basically all machines -including the management machine- need to be able to ping one each other)
• Have a user sdn that can be sudoer on the machine, without any need to insert the password for it

sudo visudo
sdn ALL=(ALL) NOPASSWD:ALL     # Add this line at the bottom of the file

• Accessing without password from the management machine to the target machines. To do this the public key of the user on the management machine must be present in the .ssh/authorized_key file on all your target machines.
  This can be done in two ways: manually or (almost) automatically using an ONOS script

ssh-keygen -t rsa         	        # If an .ssh directory is not present under your user directory, generate new SSH keys
vi ~/.ssh/authorized_keys           # Create the authorized_keys file and copy into it your management machine public key (you can find it on the managemnt machine under .ssh/id_rsa.pub)
chmod 700 ~/.ssh			        # Give the correct permissions to your .ssh directory
chmod 600 ~/.ssh/authorized_keys    # Give the correct permissions to your authorized_keys file

ssh-keygen -t rsa         	        # TBD on the management machine. If an .ssh directory is not present under your user directory, generate new SSH keys.
onos-push-keys $TARGET_MACHINE_IP   # TBD on the management machine for each target machine

• As a proof that everything works, from your management machine you should be able to login into any target machine doing ssh sdn@TARGET_MACHINE_IP and execute sudo echo hello (for example) with no passwords required

ONOS target machines

Additionally, ONOS target machines should have

• Oracle Java (at least JRE) 1.8 installed. The ONOS installation section, under the Administration Guide, provides examples for Ubuntu about how to do that.
• curl installed
• make sure the following ports are open:

  • 8181    for REST API and GUI

  • 8101    to access the ONOS CLI

  • 9876    for intra-cluster communication (communication between target machines)

  • 6653    optional, for OpenFlow

  • 6640    optional, for OVSDB

Mininet target machine

Additionally, the Mininet target machine should have

• Mininet install. This can be simple as typing sudo apt-get install mininet. Anyway, for more, and more detailed instructions, please refer directly to the Mininet website

Management/Development/build machine

Some test scenario, such as smoke, require external libraries.  (e.g. requests)  If you see errors like "ImportError: No module named requests" install them on the management machine.  

pip install requests

Test Cells

Since Lastly, since a developer may want to test different scenarios against different sets of VMs or servers, ONOS provides the notion of test cells.  

Test Cells

A test cell refers to a specific controller cluster target machines environment designated for testing. This can mean a set of bare-metal servers, or a set of VMs running on developer’s laptop.   Cell definition files are bash snippets that simply export a few environment variables into the developer’s (or tester’s) shell. These variables are then used by a number of the ONOS test and utility scripts. This allows the scripts and test scenarios to be independent to a specific test bench setup.

Using cells

.

The cell and cells commands are available on the management machine, where the ONOS repository has been originally cloned.

Listing existing cells

Cells configured can be listed using the command cells.

$ cells
beast             # Bare metal cluster (7-node)
beast-1           # Bare metal cluster (1-node)
beast-3           # Bare metal cluster (3-node)
beast-5           # Bare metal cluster (5-node)
demo              # LXC demo environment for oneping tutorial
demo-eu           # For demo with GEANT topology and three ONOS instances using vagrant devmachine
demo-eu-single    # For demo with GEANT topology and three ONOS instances using vagrant devmachine
demo-eu-single-vpls *  # For demo with GEANT topology and three ONOS instances using vagrant devmachine
ec2               # EC2-based cluster (7-node)
i2                # SDN-IP ProxMox cell
i2-centos-single  # SDN-IP ProxMox cell
i2-single         # SDN-IP ProxMox cell
local             # Local VirtualBox-based ONOS instances 1,2 & ONOS mininet box

Cell definitions are loaded into the shell environment with the cell utility. This utility takes the name of a cell definition file as the argument. For example:If the command is invoked without parameters, it will just print out the cell in use.

$ cell local               # Instruct ONOS to use cell "local"
ONOS_CELL=local
OCI=192.168.56.101
OC1=192.168.56.101
OC2=192.168.56.102
OCN=192.168.56.103
ONOS_FEATURES=openflow
ONOS_NIC=192.168.56.*

$ cell                     # Print the cell currently in use
ONOS_CELL=local
OCI=192.168.56.101
OC1=192.168.56.101
OC2=192.168.56.102
OCN=192.168.56.103
ONOS_FEATURES=webconsole,onos-api,onos-core,onos-cli,onos-openflow,onos-gui,onos-rest,onos-app-fwd,onos-app-proxyarp,onos-app-tvue
ONOS_NIC=192.168.56.*

Cell files attributes

Below it's reported a list of the basic attributes you can express and find in a cell file.cell echoes back with the key environment variables understood by the ONOS scripts:

• OCI : the default target machine node IP. This is an alias for OC1. 
• OC[1-3N] : the IP addresses of the VMs hosting ONOS instances. More OC instances may be set, if necessary.target machines. You can set as many OC instances as you want (depending on the number of ONOS target machines you want to run)
• OCN : the IP address of the VM with Mininet target machine
• ONOS_FEATURES : a comma-separated list of bundle names, loaded at startup by an ONOS instance instances within this cell
• ONOS_NIC : The address block used amongst ONOS instances for inter-controller (clustering) and OpenFlow communication communication. In the example above, it's assumed that all ONOS instances have an IP address in the subnet 192.168.56.0/24

Once a cell definition is exported, utilities such as onos onos will fall back to using the value set in OCI (192.168.56.101 for above) if not given any parameters. 

The utility script cells can be used to view all available cell definitions. Without parameters, cell will list the values associated with the current cell in useuse.

...

Defining cell files

Cell files can be found on your management machine, under $ONOS_ROOT/tools/test/cells, where $ONOS_ROOT is the onos folder containing the ONOS repository files cloned.

When a file is added to this directory, it automatically enables it to be loaded and listed using the cell and cells commands.

A cell definition file defines values for the aforementioned environment variables. A cell definition file may look like the following:

# DefaultDefining ONOSa Controllercell VMcomposed instancesby 1,2,3 ONOS target machines and aone Mininet VMtarget machine

export OC1OCI="192.168.56.101"     # Default ONOS target machine (ONOS 1)
export OC2OC1="192.168.56.102"101"     # ONOS target machine 1
export OC3OC2="192.168.56.104"102"     # ONOS target machine 2
export OCNOC3="192.168.56.103"     # Mininet VM
 ONOS target machine 1
export OCN="192.168.56.110"      
# forMininet nodetarget clusteringmachine
export ONOS_NIC=”192.168.56.*”    

# ONOS features to load
export ONOS_FEATURES="webconsole,onos-api,onos-core-trivial,onos-cli"

...

"openflow"

...

Creating custom cell definitions

Mininet topologies

Optionally, ONOS test scripts can automatically deploy also a custom Mininet topology for you on a remote Mininet target machine. Custom topologies should be defined in $ONOSCustom cell definition files may be added to ONOS_ROOT/tools/test/cells/, which enables them to be loaded and listed with cell and cells.

Test VM Setup

ONOS, being an SDN controller authored in Java, can run on a variety of platforms. However, in the interest of focus, the ONOS team engages primarily in testing on Ubuntu server distributions, specifically Ubuntu Server 14.04* LTS 64-bit.

Additionally, the VMs used for running ONOS instances and Mininet can be configured to better mesh with the functionality of the ONOS utility scripts and the test environment:

• Adding two network interfaces : One should be configured to use the host adapter and the other to use either the NAT or Bridged adapter to allow outside access.
• Adding an user named 'sdn' : The scripts check for the environment variable ONOS_USER, and, if it is unset, uses sdn as the default username. 
• Allowing password-less (key-based) login : The scripts rely on ssh and scp for their functions, and this lets one avoid having to type a password each time a script is run.

For password-less login, the onos-push-keys  utility can be used to transfer one's SSH key to the VM.

Using the Test Environment

Once set up, changes to the codebase can be (relatively) quickly tested in a distributed setting as follows:

1. Make changes to the code
2. Build with mvn clean install
3. Load the cell settings with cell <your_cell_name>
   
4. Package executables for deployment with onos-package 
5. Deploy to test VMs in the cell

For step 5, a developer can take advantage of the OC variables when using onos-install:

$ onos-install -f $OC1  #install ONOS to OC1

topos.

Listing existing topologies

To view the list of the existing topologies with their descriptions, you can use the command topos. An asterisk may indicate what topology is currently in use. An example is reported below.

$ topos
default *                  # Default US MPLS topology recipe (currently in use)
geant                      # GEANT & Nordnet topology recipe
sdnip                      # SDN-IP topology recipe
uk                         # Simple UK topology recipe
vpls                       # Default VPLS topology recipe

Loading topologies

Topologies can be load with the command topo $TOPO_NAME. The same command without arguments shows the details about the topology loaded.

$ topo default             # Load the topology named default
ONOS_TOPO=default
OTH=25
OTL=140
OTD=25

$ topo                     # Print the details about the topology currently loaded
ONOS_TOPO=default
OTH=25
OTL=140
OTD=25

Creating custom topologies

The folder already contains lots of examples that may help you to start to write your first own topology. A topology may include up to 3 files, by convention called the same way, but using different extensions:

• The Topology recipe file (.recipe). The topology recipe file lets you define a new topology. This is mandatory, in order to create a new topology. Optionally, it lets you also automatically check (after the virtual network has been deployed) if the the expected number of devices, links and hosts are set. The structure of this file is very simple. An commented example is reported below.
   
• The Mininet Python topology file, (.py) that expresses how the Mininet topology should look like. The file is mandatory if you want to deploy a Mininet topology as well (to create new topologies look at the existing examples shipped with ONOS and/or go on the Mininet website)
  
• The ONOS network configuration file (.json), that lets you specify an ONOS. You can refer to this guide to learn more about the ONOS network configuration.
  

ONOS_TOPO=default  # A new topology named "default" is declared
export OTD=6       # 6 devices are expected to be found (optional)
export OTL=18      # 18 links are expected to be found (optional)
export OTH=6       # 6 hosts are expected to be found (optional)

Deploy ONOS and the topology STC

The STC command lets you deploy multiple ONOS instances, the Mininet topology and test the environment.

Once your cell file is loaded:

$ stc setup
$ stc setup
2017-01-27 15:11:52  Setup started
2017-01-27 15:11:52  Push-Bits-1 started -- onos-push-bits 10.100.198.201
2017-01-27 15:11:52  Uninstall-1 started -- onos-uninstall 10.100.198.201
2017-01-27 15:11:55  Push-Bits-1 completed
2017-01-27 15:12:11  Uninstall-1 completed
2017-01-27 15:12:11  Kill-1 started -- onos-kill 10.100.198.201
2017-01-27 15:12:11  Kill-1 completed
2017-01-27 15:12:11  Install-1 started -- onos-install 10.100.198.201
2017-01-27 15:12:22  Install-1 completed
2017-01-27 15:12:22  Secure-SSH-1 started -- onos-secure-ssh -u onos -p rocks 10.100.198.201
2017-01-27 15:12:38  Secure-SSH-1 completed
2017-01-27 15:12:38  Wait-for-Start-1 started -- onos-wait-for-start 10.100.198.201
2017-01-27 15:12:43  Wait-for-Start-1 completed
2017-01-27 15:12:43  Check-Components-1 started -- onos-check-components 10.100.198.201
2017-01-27 15:12:43  Check-Nodes-1 started -- onos-check-nodes 10.100.198.201
2017-01-27 15:12:46  Check-Nodes-1 completed
2017-01-27 15:12:48  Check-Components-1 completed
2017-01-27 15:12:48  Check-Logs-1 started -- onos-check-logs 10.100.198.201
2017-01-27 15:12:48  Check-Apps-1 started -- onos-check-apps 10.100.198.201 drivers,openflow includes
2017-01-27 15:12:49  Check-Logs-1 completed
2017-01-27 15:12:49  Check-Apps-1 completed
2017-01-27 15:12:49  Setup completed
0:56 Passed! 11 steps succeeded

$ stc net-setup
2017-01-27 15:13:34  Net-Setup started
2017-01-27 15:13:34  Push-Topos started -- onos-push-topos 10.100.198.200
2017-01-27 15:13:34  Stop-Mininet-If-Needed started -- onos-mininet stop
2017-01-27 15:13:34  Push-Topos completed
2017-01-27 15:13:35  Stop-Mininet-If-Needed completed
2017-01-27 15:13:35  Clean-Mininet-If-Needed started -- onos-mininet cleanup
2017-01-27 15:13:37  Clean-Mininet-If-Needed completed
2017-01-27 15:13:37  Wipe-Out-Data-Before started -- onos-wipe-out
2017-01-27 15:13:37  Wipe-Out-Data-Before completed
2017-01-27 15:13:37  Initial-Summary-Check started -- onos-check-summary 10.100.198.201 [0-9]* 0 0 0
2017-01-27 15:13:37  Initial-Summary-Check completed
2017-01-27 15:13:37  Config-Topo started -- onos-netcfg 10.100.198.201 /Users/luca/onos/tools/test/topos/default.json
2017-01-27 15:13:38  Config-Topo completed
2017-01-27 15:13:38  Start-Mininet started -- onos-mininet start topos/topo default.py 10.100.198.201
2017-01-27 15:13:38  Start-Mininet completed
2017-01-27 15:13:38  Wait-For-Mininet started -- onos-mininet wait 10
2017-01-27 15:13:56  Wait-For-Mininet completed
2017-01-27 15:13:56  ARP-Hosts started -- onos-mininet sendAndExpect gratuitousArp --expect .
2017-01-27 15:13:59  ARP-Hosts completed
2017-01-27 15:13:59  Check-Summary started -- onos-check-summary 10.100.198.201 [0-9]* 25 140 25
2017-01-27 15:13:59  Check-Summary completed
2017-01-27 15:13:59  Check-Flows started -- onos-check-flows 10.100.198.201
2017-01-27 15:14:04  Check-Flows completed
2017-01-27 15:14:04  Net-Setup completed
0:30 Passed! 12 steps succeeded

Putting everything together: using the test environment

Let's wrap up! The beauty of the cell system is that it will give you the flexibility to test different ONOS configurations on different Mininet topologies, on different remote machines (regardless they are physical servers, virtual machines, containers, ...).

A typical workflow may look like the following:

1. Make changes to the code 
2. Build ONOS with onos-buck buld onos (more details here)
3. Load the cell settings with cell $YOUR_CELL_NAME
   
4. Load your topology with topo $YOUR_TOPO_NAME
   
5. Deploy ONOS target machines with stc setup
6. Deploy the Mininet topology with stc net-setup
   

The correspondent one line command and the related output is reported below:

$ cell $YOUR_CELL_NAME && topo $YOUR_TOPO_NAME && stc setup && stc net-setup   
  

Other useful commands

Attach to the remote ONOS machine OCN

$ onos-mininet attach

This procedure must be repeated for each target, as onos-install is only capable of handling one target at a time. In ONOS 1.1.0 and later, the onos-group command can be used to automatically target all cell member VMs at once:Start/Stop/Restart/Check the status of the ONOS service

$ onos-groupservice install -f-cell start/stop/restart/status    # instead of cell you can also specify $OCX

Utilities such as onos-service also take the --cell argument that enables it to manage all cell members at once. For example, to restart all ONOS instances in a cell:Kill the ONOS process

$ onos-service --cell restart

Using Lightweight Virtualization

Finally, a developer may not be able to run multiple VMs due to the resource constraints of their build machine or environment. In such cases, lightweight virtualization technologies may be used to supply the resource containment for each ONOS instance. Multiple instances using LXC demonstrates the use of LXC containers for this purpose.

Previous : Development Environment Setup
Next : Continuous Integration

...

start/stop/restart    # instead of cell you can also specify $OCX