...

Running multiple VMs that each run an ONOS instance is one way of running a multi-instance ONOS deployment. It is however not practical to do on my resource-constrained laptop. Using Linux Containers is a great alternative that achieves the same thing but uses way less CPU and memory. I also didn't care much for Docker's approach: Docker is very convenient to distribute ONOS as a self-contained application, but I had trouble integrating it into my development cycle which requires fast code/deploy/test cycles.

Getting Started

I'm using VirtualBox as virtualization environment, but I see no reason why this shouldn't work on VMWare or other hypervisors.

...

This VM will eventually host the ONOS instance containers. In the following, we'll first create a single container that is fully configured, and then clone the original as many times as needed.

Using Vagrant boxes to prepare your development environment

This section replaces the remainder of this document and is an alternative to bootstrapping your environment. Using vagrant we provision two VMs:

...

Enjoy and feel free to report any issues.

Creating Your First Container

Installing LXC is as simple as running

...

Take a look at the output of lxc-ls again: note the container is now started and, if all has gone well, has received an IP address.

Cell Configuration

Containers are very convenient since we can automatically obtain their IPs. This is what my cell definition looks like; you want to put this in a file in onos/tools/test/cells/. You definitely want to verify if the ONOS_NIC variable corresponds to your system settings, and also OCN which points to the machine where Mininet is running. Feel free to customize the ONOS_APPS to your liking. Finally, you have to reload the cell if you create additional containers, so the new IPs get picked up.

# LXC-based multi ONOS instance & LINC-OE mininet box
export ONOS_NIC=10.0.3.*
I=1
for CONTAINER in $( sudo lxc-ls ); do
 IP=`sudo lxc-info -n $CONTAINER -iH`
 export OC${I}=${IP}
 let I=I+1
done
export OCI=$OC1
export OCN="192.168.56.9"
export ONOS_APPS="drivers,openflow,proxyarp,optical,bgprouter"
export ONOS_USER=ubuntu
export ONOS_GROUP=ubuntu

Customizing Your Container

You want to do three things: enable paswordless login, enable passwordless sudo, and install Java 8.

...

Finally, log out of the container by pressing Ctrl-d or typing exit.

Cloning Your Container

We're now ready to make as many clones of the original container as you want ONOS instances. Repeat the following steps as many times as you'd like, giving each new container a unique name.

...

Be sure to reload your cell configuration, which will automatically create and assign new $OCx variables.

Development Cycle

From now on, it becomes very easy to develop and test code in a multi-instance environment. The process is as follows:

1. Write some code
2. Compile ONOS by running obs or mci
3. Package ONOS by running op
4. Install and run the latest package: onos-group install -f $OC1 $OC2 $OC3 etc.
5. Go back to step 1 

Additional Steps

Linux Bridge Configuration

I ran into the weird problem that the ONOS instances weren't discovering each other. This process is driven by Hazelcast and relies on IP multicast. It turned out that LXC uses Linux bridge for connectivity among the containers, and the bridge by default does not forward multicast traffic! This is easily solved by checking the Linux bridge documentation; here's an excerpt

...

sudo echo 2 > /sys/devices/virtual/net/lxcbr0/bridge/multicast_router

Start Containers on Boot

If you would like your containers to be automatically started on boot, you'll need to add the following line to /var/lib/lxc/NAME/config, where NAME is your container's name. By the way, on my system the /var/lib/lxc directory is only accessible by root.

...

Check using lxc-ls --fancy to verify your containers are in fact started on boot.

Port Forwarding

If you want to access the ONOS GUI from a browser on the host machine, you need to set up port forwarding on the VM. The following command forwards traffic on port 8181 to the first ONOS container ($OC1).

...