Environment setup with Mininet and onos.py

Mininet is a lightweight container orchestration system for network emulation. With Mininet and onos.py, you can easily start up an ONOS cluster, and a modeled data network for any topology you might like, in a single VM or server. This is usually the most convenient way to create an ONOS development environment on your laptop, and you can be up and running in a matter of minutes (or seconds if you have already built ONOS and have already installed Mininet!)

Why use onos.py and Mininet?

If you are already running Mininet in a VM or on a physical server, it is easy to use onos.py to start up a complete emulated ONOS network in a single VM - including a ONOS cluster, modeled control network, and data network.

This simplifies development on a laptop, because you can run a single development VM (or no VM at all if you are running on a Linux machine!) Moreover, it is more efficient than a multi-VM setup because the entire emulated network lives in a single VM and shares a single Linux kernel.

Additionally, onos.py models the control network as well as the data network; you can easily change the number of nodes in your ONOS cluster, as well as things like the delay or bandwidth between nodes in the control network. It's even possible to change the control network topology as well as the data network topology. (We hope to make this more convenient and powerful in the future.)

onos.py provides a single, unified console via the mininet-onos> prompt, where you can enter both Mininet and ONOS commands - this can be very convenient! (We call this "One Console to Rule Them All.")

onos.py also automatically handles port forwarding, so you can easily connect to the GUI (or to karaf, or to the controllers' OpenFlow ports) by connecting to ports on the VM.

Chances are you're already using Mininet, so it's nice to be able to start an ONOS cluster using Mininet itself without installing or configuring additional software.

onos.py parametrizes both the control network (ONOS cluster) and the data network; so it's easy to iterate over multiple cluster sizes and network topologies.

We also hope that using onos.py will make ONOS development easier and more fun.

Setting up a development VM or server

This document assumes that you already have a VM or server  (we recommend Ubuntu 16 LTS) where you can compile ONOS, preferably using buck (or possibly with Maven - see below.)

Getting Mininet

In order to use onos.py, you need Mininet, which is easy to install in a VM or server running a recent release of Ubuntu.

Option 1: If you are running Ubuntu 16.04, you can easily install Mininet 2.2 (and bridge-utils for LinuxBridge) using apt-get:

apt-get install mininet bridge-utils

Option 2: It's also easy to install Mininet from source in an Ubuntu VM or server:

git clone http://github.com/mininet/mininet
mininet/util/install.sh -nvfw

(recommended options: -n: install core Mininet dependencies; -v: install Open vSwitch; -f: install legacy Stanford 1.0 user switch and controller; -w: install wireshark and improved OpenFlow wireshark dissector)

Option 3: You can also download a pre-built Mininet VM.

Verifying Your Mininet Installation

Make sure you are running Mininet version 2.2.1 or later:

mn --version

If not, try installing from the git repository as described above.

And make sure that Mininet works:

sudo mn --test pingall

If Mininet doesn't work, consult the troubleshooting section below.

Building ONOS

Make sure that you have built ONOS in your Mininet VM or server using buck:

cd ~/onos
buck build onos

Make sure that the build completes without errors - without a correct ONOS package, you will not be able to start ONOS.
 

cd ~/onos

mvn clean install  # or mci for short; needs to complete without errors

onos-package  # once again, needs to complete without errors

cd ~/onos/tools/dev/mininet

# See below for more details on options for mn

sudo env ONOS_TAR=/tmp/onos-1.7.0.openflow.tar.gz mn --custom onos.py ...

sudo file /tmp/onos-1.7.0.openflow.tar.gz

/tmp/onos-1.7.0.openflow.tar.gz: gzip compressed data, from Unix, last modified: Thu Jun  2 15:08:11 2016

Starting up ONOS and your data network

Next, use Mininet and onos.py to start up a virtual ONOS cluster and data network:

cd ~/onos/tools/dev/mininet
sudo mn --custom onos.py --controller onos,1 --topo tree,2,2

(In this example, our "cluster" is a single node, but we will change that in a minute!)

You should see a bunch of output showing the startup of the control network and the data network.

After ONOS starts up and the switches connect, you should see the customized mininet CLI prompt:

mininet-onos>

At this point, you can enter mininet commands like pingall (all-to-all ping test) or help (to find out about Mininet CLI commands.)

You can also enter ONOS commands like onos:apps or onos:balance-masters and they should be invoked via karaf's client command on onos1.

You can also invoke the ONOS client using the onos command - press control-D to exit.

You can also connect to the ONOS GUI by following the instructions below.

To exit Mininet, use the exit command or press control-D.

Once you have verified that a single-node ONOS cluster and simple topology are working correctly, you can try a larger cluster (we recommend 3 nodes if you have configured your VM to use 6-12 GB of RAM) and a larger or more interesting topology (such as a 4x4 torus):

sudo mn --custom onos.py --controller onos,3 --topo torus,4,4

Questions and Answers

What do those mn command line options do?

--custom onos.py: uses onos.py to extend Mininet with new controller and switch types as well as a customized CLI

--controller onos,3: tells Mininet to start up an ONOS controller cluster with 3 ONOS nodes

--topo torus,4,4: tells Mininet to use a 4x4 torus topology for the data network

What software switch does this use?

By default, it uses Open vSwitch. onos.py replaces Mininet's default switch with a new switch class called ONOSOVSSwitch or --switch onosovs. This switch class knows how to connect to an ONOS cluster with multiple IP addresses.

How can I use the user switch (or CPqD switch if I have it installed?)

In order to select the user switch (either Stanford reference switch or CPqD switch, depending on which one you have installed), you can use --switch onosuser.

How can I get more information on Mininet, the mn command, writing Mininet scripts, etc.?

There are lots of things you can do with Mininet, including customizing your data network topology, setting link parameters, etc..

For more information on Mininet, please check out http://docs.mininet.org

How can I specify the apps for ONOS to load?

For now, try using ONOS_APPS or connecting to the karaf console and using ONOS CLI commands. As root:

ONOS_APPS=drivers,openflow,fwd,proxyarp,mobility mn --custom onos.py --controller onos,3 --topo tree,3,3

or (with sudo):

sudo env ONOS_APPS=drivers,openflow,fwd,proxyarp,mobility mn --custom onos.py --controller onos,3 --topo tree,3,3

or

ONOS_APPS=drivers,openflow,fwd,proxyarp,mobility sudo -E mn --custom onos.py --controller onos,3 --topo tree,3,3
 

In the future, there should be an option to --controller onos and/or ONOSCluster().

How is the IP address range/subnet of the ONOS cluster specified?

--controller onos and ONOSCluster() take an ipBase option; the default is 192.168.123.0/24

You can change it by passing in a new option to mn or to ONOSCluster():

sudo mn --custom onos.py --controller onos,3,ipBase=172.1.2.0/24 --topo tree,3,3

How can I connect to the ONOS console from the mininet-onos> prompt?

mininet-onos> onos

press control-D to return to the mininet-onos> prompt.

How can I run ONOS/karaf CLI commands from the mininet-onos> prompt?

There are several ways of doing it - you can use the one which works best for you. Here are four equivalent ways of invoking the onos:apps command:

mininet-onos> onos:apps

mininet-onos> onos :apps

mininet-onos> onos onos:apps

mininet-onos> onos1 client onos:apps

The first two methods can be used for executing any commands that begin with "onos:".

The last two methods can be used to execute any karaf command.

The final method can be used to run client on a specific ONOS instance (and connect to that instance.)

How can I connect to the ONOS console from a shell/bash prompt?

Make sure karaf's bin/ directory is in your path, then:

client -h 192.168.123.1   # or the address of the ONOS node you wish to connect to  

How can I connect to the karaf console using ssh?

ssh -p 8101 karaf@192.168.123.1

In this example, 8101 is the karaf port, and 192.168.123.1 is the IP address of onos1.

Note that this is using ssh inside the Mininet VM itself - from another machine, you should use the VM's IP address and the appropriate forwarded port as described below.

How can I connect to the ONOS GUI if ONOS is running in a Mininet host inside a VM?

One way is to run a browser inside the VM. Install a GUI in your VM if you haven't already (sudo apt-get install lxde and reboot), and then run a web browser pointed to the IP address of the ONOS node you want to talk to (for onos1, the default will be http://192.168.123.1/8181/onos/ui ). For the GUI to work well, you will want to make sure you've installed the appropriate VM support tools (such as virtualbox tools or VMware tools) for your virtualization platform.

Another (perhaps more enjoyable) way is to run a browser on your client machine or server where the VM is running, as long as you have connectivity to the VM (make sure you can ping its IP address and/or connect to it using ssh, and make sure that it the ports on your VM aren't being blocked by a firewall.)

By default, onos.py automatically forwards connections to the Mininet VM/server to the appropriate ONOS instance.

So if you are using a VM whose IP address is 192.168.x.y, to connect to the GUI on ONOS1, you would use the URL http://192.168.x.y:8181/onos/ui/

If you are using VirtualBox, you will probably need to add a host-only interface to your VM and you may even need to run dhclient to make sure that that interface has an IP address. You can also set up port forwarding on a single interface - in this case you will want to make sure that it is set up for every port you wish to connect to (e.g. 8181 for the GUI on onos1.)

What are the default user name and password for the GUI?

Usually karaf/karaf (or sometimes onos/rocks)

onos.py should also respect the values of ONOS_WEB_USER and ONOS_WEB_PASS.

Can I use onos.py to control an external hardware (or virtual) network?

Yes you can! Simply use --topo none ; for example,

mn --custom onos.py --controller onos,3 --topo none

onos.py will forward the ports (e.g. 6653, 6654, 6655) appropriately as described above. Your Mininet VM or server will have to be reachable from the switches' management network, and you will need to configure your switches to connect to you ONOS nodes: the address of each node will be the VM's IP address, and the port will be the appropriate forwarded OpenFlow port.

Troubleshooting

onos.py doesn't work - what should I do?

Here are some things you can try:

• Make sure you've given your VM enough RAM - try using 2-4 GB for each ONOS node (so 6-12 GB for --controller onos,3)
• Make sure you are running the latest version of Mininet (see below)
• Look at the log file(s) and make sure no errors are occurring: more /tmp/onos1/log; more /tmp/onos1/onos.log
• Look at the output from running mn or onos.py and carefully examine and try to understand any error or exception messages - usually they will tell you what is going wrong
• Reset your environment using: sudo pkill -f  karaf.jar; sudo mn -c
• Make sure you have built onos: cd ~/onos; buck build onos 
• Make sure you can run Mininet: sudo mn --test pingall 

• Invoke Mininet in debug mode by adding the -v debug flag to see what is going wrong or hanging:
  • sudo mn --custom onos.py --controller onos,1 --topo tree,2,2 -v debug
  • You will want to pay careful attention to each operation as well as any error messages that are being displayed or exceptions that may be occurring.

Mininet doesn't work - what should I do?

First, make sure you are running Mininet 2.2.1 or later:

mn --version

If you have an older version of Mininet, you can install a newer version from source as described above. Older versions of Mininet may not include LinuxBridge, which is required by onos.py.

Next, make sure that the following command works:

sudo mnexec -n ifconfig -a

It should show a single loopback interface and no other interfaces. If this command doesn't work as expected, then Mininet will not work. If you have installed Mininet using apt-get, or from source on Ubuntu, Mininet should work correctly. However, you should make sure you don't have multiple or older versions of Mininet installed which might not work correctly with onos.py.

Once you've verified that mnexec works, you can try a sanity check for mn:

sudo mn --test pingall

This should complete without errors. If it doesn't, try invoking Mininet in debug mode so that you can see what isn't working:

sudo mn --test pingall -v debug

Why can s1 ping onos1, but onos1 can't ping s1 (it just pings its own loopback interface?)

Currently, all of the switches are in the root namespace. If you wish to test connectivity from a switch to a controller, try pinging nat0 instead. In the future we may add support for a more complex control network with separate IP addresses for the switch management interfaces.

Why is ONOS startup so slow?

On my laptop, a 3-node ONOS cluster typically takes about a minute to start up, and a single-node ONOS cluster takes up to 20 seconds to start up.

We hope to improve ONOS cluster startup performance in the future.

If your VM or server isn't heavily loaded but startup seems to be frozen for more than a couple of minutes, you can press control-C to interrupt onos.py startup and then:

sudo pkill java  # get rid of stale java processes
sudo mn -c  # clean up stale Mininet state and interfaces

You can also look at /tmp/onos1/log to see if exceptions are occurring.

Why can't I connect to the GUI that is running somewhere inside my VirtualBox VM?

First, make sure that you've configured your VM appropriately to enable connectivity:

• Usually this means adding a host-only interface (and possibly configuring it with dhclient)
• Make sure that you know the correct IP address of the interface you're connecting to
• Make sure you can ping that interface from your client machine
• If you're using port forwarding in VirtualBox, double-check to make sure that you've set it up correctly and are connecting to the correct IP address
• Make sure that ONOS is running and that the GUI module is loaded and running
  • You can check this with the onos:apps --active  CLI command