The content of this page is for users who already have some experience with ONOS and Trellis and wish to learn how to use those with P4 devices. If you are getting started with ONOS and Trellis we suggest you start with the following slides and exercises: Trellis+P4 Tutorial (Presented at ONF Connect, Dec 2018): http://bit.ly/trellis-p4-slides. |
Before starting, make sure to have ONOS on your physical machine. To download, build and run ONOS, follow this guide: Developer Quick Start
Make also sure that Docker is installed and you can run Docker containers in your machine (Install Docker).
In this guide we will refer to ${WORKSPACE} as the environment variable of your workspace, where we suppose also ONOS has been downloaded. Define it on your .bashrc file or make sure that is defined in all your terminal windows.
We provide a convenient Docker container with Mininet and BMv2 already installed (P4mn on Docker Hub).
To pull the container, type the following command in a terminal window on your machine:
$ docker pull opennetworking/p4mn |
To try fabric.p4 with BMv2, you will need a custom Mininet script distributed as part of the Trellis "routing" repository:
To clone the repository, type the following commands in a terminal window on your machine:
cd ${WORKSPACE}
git clone https://github.com/opennetworkinglab/routing/ |
Reference: https://wiki.opencord.org/pages/viewpage.action?pageId=3014916
In Trellis, we run every switch as router, each router needs a mac address and multiple interface configurations.
Interface config:
"Device-id/port-number" : {
"interfaces" : [
{
"name": "interface name",
"ips" : [ "10.0.1.254/24" ],
"vlan-untagged": 10
}
]
} |
Name: Interface name, can be empty or unset;
ips: List of IP addresses for this interface, it can be the default gateway of the host;
VLAN config, it can be: vlan-untagged only, vlan-tagged only or vlan-tagged + vlan-native.
vlan-untagged (integer): this interface consumes untagged packet only and it belongs to a specific vlan;
vlan-tagged (integer array): this interface can consumes specific tagged vlans;
vlan-native (integer): if an untagged consumes by this port, and this is an interface with vlan-tagged setup, device will use this vlan for internal use;
Example of Segment Routing device settings:
"Device-id" : {
"segmentrouting" : {
"name" : "Device name",
"ipv4NodeSid" : 123,
"ipv4Loopback" : "192.168.0.123",
"ipv6NodeSid" : 223,
"ipv6Loopback" : "2000::c0a8:0223",
"routerMac" : "00:00:00:00:01:23",
"isEdgeRouter" : true,
"adjacencySids" : []
}
} |
Name: Name for segment routing device;
ipv4(6)NodeSid: Global unique id for IP loopback, it is used as MPLS label in forwarding;
ipv4(6)Loopback: Router loopback IP address, it should not be part of same same subnet defined on dataplane interface;
routerMac: Globally unique Mac address. It will be used to reply ARP requests for router IP or interface IP;
isEdgeRouter: true if this device is edge/leaf, otherwise false;
adjacencySids: Reserved, put empty array for now.
In the demo Mininet topology, we set the gateway of the hosts to the IP of the interface of the leaf switch. For example, the gateway of Host 1 will be 10.0.2.254.
In this tutorial, we need 3 terminal windows for:
ONOS karaf process (from now on it will be called T1)
ONOS shell (T2)
Docker P4+Mininet emulator (T3)
So, first of all, open 3 terminal windows on your machine.
In T1 run ONOS with the following commands:
$ cd ${WORKSPACE}/onos
$ ONOS_APPS=drivers,openflow,netcfghostprovider,segmentrouting,fpm,dhcprelay,routeradvertisement,t3,hostprobingprovider,drivers.bmv2,pipelines.fabric
$ bazel run onos-local -- clean |
In T2 connect to the ONOS shell running the following command:
$ onos localhost |
Now check that the apps are correctly active. You should see something similar to the output shown below:
onos> apps -a -s * 9 org.onosproject.drivers 1.13.0.SNAPSHOT Default Drivers * 35 org.onosproject.generaldeviceprovider 1.13.0.SNAPSHOT General Device Provider * 36 org.onosproject.protocols.grpc 1.13.0.SNAPSHOT gRPC Protocol Subsystem * 37 org.onosproject.protocols.p4runtime 1.13.0.SNAPSHOT P4Runtime Protocol Subsystem * 38 org.onosproject.p4runtime 1.13.0.SNAPSHOT P4Runtime Provider * 39 org.onosproject.drivers.p4runtime 1.13.0.SNAPSHOT P4Runtime Drivers * 42 org.onosproject.proxyarp 1.13.0.SNAPSHOT Proxy ARP/NDP * 44 org.onosproject.hostprovider 1.13.0.SNAPSHOT Host Location Provider * 45 org.onosproject.lldpprovider 1.13.0.SNAPSHOT LLDP Link Provider * 73 org.onosproject.pipelines.basic 1.13.0.SNAPSHOT Basic Pipelines * 84 org.onosproject.route-service 1.13.0.SNAPSHOT Route Service Server * 95 org.onosproject.protocols.gnmi 1.13.0.SNAPSHOT gNMI Protocol Subsystem * 96 org.onosproject.drivers.gnmi 1.13.0.SNAPSHOT gNMI Drivers * 108 org.onosproject.pipelines.fabric 1.13.0.SNAPSHOT Fabric Pipeline * 120 org.onosproject.segmentrouting 1.13.0.SNAPSHOT Segment Routing * 121 org.onosproject.drivers.bmv2 1.13.0.SNAPSHOT BMv2 Drivers |
From T3, now push the network configuration:
$ cd ${WORKSPACE}/routing/trellis
$ onos-netcfg localhost trellisp4.json |
Now ONOS is running, the needed apps are correctly running and the network configuration is pushed in ONOS, we are ready to run the emulated trellis topology on P4+Mininet Docker container.
In T3 start the previously downloaded Docker container with the topology available in the "routing" repository. You can do it running the following command:
$ docker run --rm --privileged -v /tmp/p4mn:/tmp -v${WORKSPACE}/routing:/routing -w/routing/trellis -it --name p4trellis --hostname p4trellis -p 50001-50030:50001-50030 --env PYTHONPATH=/root --entrypoint python opennetworking/p4mn:stable trellisp4.py --onos-ip 127.0.0.1 |
Start mininet topology on new terminal using trellisp4.py python script
trellisp4.py creates:
2 by 2 leaf-spine topology using 4 bmv2 devices
4 bidirectional links between leaf and spine
4 hosts, 2 hosts per leaf device
$ sudo -E env PYTHONPATH=$PYTHONPATH:$ONOS_ROOT/tools/dev/mininet ./trellisp4.py --onos-ip 127.0.0.1; |
After started, ONOS should discover 4 devices and 8 links
onos> devices -s id=device:bmv2:204, available=true, role=MASTER, type=SWITCH, driver=bmv2:org.onosproject.pipelines.fabric id=device:bmv2:205, available=true, role=MASTER, type=SWITCH, driver=bmv2:org.onosproject.pipelines.fabric id=device:bmv2:226, available=true, role=MASTER, type=SWITCH, driver=bmv2:org.onosproject.pipelines.fabric id=device:bmv2:227, available=true, role=MASTER, type=SWITCH, driver=bmv2:org.onosproject.pipelines.fabric onos> links src=device:bmv2:204/1, dst=device:bmv2:226/1, type=DIRECT, state=ACTIVE, expected=false src=device:bmv2:204/2, dst=device:bmv2:227/1, type=DIRECT, state=ACTIVE, expected=false src=device:bmv2:205/1, dst=device:bmv2:226/2, type=DIRECT, state=ACTIVE, expected=false src=device:bmv2:205/2, dst=device:bmv2:227/2, type=DIRECT, state=ACTIVE, expected=false src=device:bmv2:226/1, dst=device:bmv2:204/1, type=DIRECT, state=ACTIVE, expected=false src=device:bmv2:226/2, dst=device:bmv2:205/1, type=DIRECT, state=ACTIVE, expected=false src=device:bmv2:227/1, dst=device:bmv2:204/2, type=DIRECT, state=ACTIVE, expected=false src=device:bmv2:227/2, dst=device:bmv2:205/2, type=DIRECT, state=ACTIVE, expected=false |
Hosts should be learned by ONOS
onos> hosts id=00:AA:00:00:00:01/None, mac=00:AA:00:00:00:01, locations=[device:bmv2:204/3], vlan=None, ip(s)=[10.0.2.1], provider=of:org.onosproject.provider.host, configured=false id=00:AA:00:00:00:02/None, mac=00:AA:00:00:00:02, locations=[device:bmv2:204/4], vlan=None, ip(s)=[10.0.2.2], provider=of:org.onosproject.provider.host, configured=false id=00:AA:00:00:00:03/None, mac=00:AA:00:00:00:03, locations=[device:bmv2:205/3], vlan=None, ip(s)=[10.0.3.1], provider=of:org.onosproject.provider.host, configured=false id=00:AA:00:00:00:04/None, mac=00:AA:00:00:00:04, locations=[device:bmv2:205/4], vlan=None, ip(s)=[10.0.3.2], provider=of:org.onosproject.provider.host, configured=false |
If any host was not discovered by ONOS, try send arp request by using arping tool from mininet shell:
mininet> h1 arping 10.0.2.254 |
Routing and bridging rule should be installed by ONOS after host been detected and hosts should be able to ping each other:
mininet> pingallfull <...skip debug logs...> *** Results: h1->h2: 1/1, rtt min/avg/max/mdev 1.267/1.267/1.267/0.000 ms h1->h3: 1/1, rtt min/avg/max/mdev 3.384/3.384/3.384/0.000 ms h1->h4: 1/1, rtt min/avg/max/mdev 4.708/4.708/4.708/0.000 ms h2->h1: 1/1, rtt min/avg/max/mdev 1.381/1.381/1.381/0.000 ms h2->h3: 1/1, rtt min/avg/max/mdev 3.138/3.138/3.138/0.000 ms h2->h4: 1/1, rtt min/avg/max/mdev 3.983/3.983/3.983/0.000 ms h3->h1: 1/1, rtt min/avg/max/mdev 3.446/3.446/3.446/0.000 ms h3->h2: 1/1, rtt min/avg/max/mdev 3.315/3.315/3.315/0.000 ms h3->h4: 1/1, rtt min/avg/max/mdev 1.388/1.388/1.388/0.000 ms h4->h1: 1/1, rtt min/avg/max/mdev 3.345/3.345/3.345/0.000 ms h4->h2: 1/1, rtt min/avg/max/mdev 4.479/4.479/4.479/0.000 ms h4->h3: 1/1, rtt min/avg/max/mdev 1.302/1.302/1.302/0.000 ms mininet> |