...
Note that this instructions assume you’re familiar with ONOS and OpenStack, and do not provide a guide to how to install or trouble shooting these services. However, If you aren’t, please find a guide from ONOS(http://wiki.onosproject.org) and OpenStack(http://docs.openstack.org), respectively.
The example deployment depicted in the above figure uses three networks with an external router.
- Management network: used for ONOS to control virtual switches, and OpenStack to communicate with nova-compute agent running on the compute node
- Data network: used for East-West traffic via VXLAN, GRE or GENEVE tunnel
- External network: used for North-South traffic, normally only gateway nodes have an access to this network
All networks can share a network interface in case your test machine does not have enough interfaces. You can also emulate external router. The figure below shows an example test environment used in the rest of this guide with emulated external router and two network interfaces, one for sharing management and external, and the other for data.
Prerequisite
1. Install OVS to all nodes including compute and gateway. Make sure your OVS version is 2.3.0 or later (later than 2.5.0 is recommended). Refer to this guide for updating OVS (don't forget to change the version in the guide).
Note |
---|
In ONOS version 1.11 (Loon) or later, stateful NAT feature is included, which is disabled by default. If you want to use the stateful NAT feature, you have to install OVS 2.6 or later. |
2. Set OVSDB listening mode. Note that "compute_node_ip" in the below command should be an address accessible from the ONOS instance.
...
Code Block |
---|
set "$@" --remote=ptcp:6640 |
...
Now you should be able to see port "6640" is in listening state.
...
Code Block | ||
---|---|---|
| ||
$ sudo ovs-vsctl show cedbbc0a-f9a4-4d30-a3ff-ef9afa813efb ovs_version: "2.58.02" |
OpenStack Setup
How to deploy OpenStack is out of scope of this documentation. Here, it only describes configurations related to use SONA. All other settings are completely up to your environment.
Controller Node
1. Install networking-onos (Neutron ML2 plugin for ONOS) first.
Note |
---|
The guide is based on OpenStack Ocata/Pike version. If you want to install Newton version of OpenStack, please refer to here <TBA>. |
1. The first step is installing networking-onos. Please create '/opt/stack' folder and install networking-onos in /opt/stack as follows.
Code Block | ||
---|---|---|
/opt/stack$ git clone --branch [stable/ocata or stable/pike] | ||
Code Block | ||
| ||
$ git clone https://github.com/openstack/networking-onos.git $ pip install ./networking-onos |
Next, please create a file /opt/stack/2. Edit ml2_conf_onos.ini uner networking-onos/etc/neutron/plugins/ml2 for ONOS endpoint and credential. You may want to copy the config file under /etc/neutron/plugins/ml2/ where other Neutron configuration files areconf_onos.ini using the following template. Please set the IP_ADDRESS_OF_ONOS to host IP address of ONOS controller.
Code Block | ||||
---|---|---|---|---|
| ||||
# Configuration options for ONOS ML2 Mechanism driver [onos] # (StrOpt) ONOS ReST interface URL. This is a mandatory field. url_path = http://10.134.231.29IP_ADDRESS_OF_ONOS:8181/onos/openstacknetworking # (StrOpt) Username for authentication. This is a mandatory field. username = onos # (StrOpt) Password for authentication. This is a mandatory field. password = rocks |
Note |
---|
URL path is changed from "onos/openstackswitching" to "onos/openstacknetworking" since 1.8.0. |
3. Next 2. The next step is installing and running OpenStack services. For DevStack users, use the following sample DevStack local.conf to build OpenStack controller node. Make sure your DevStack branch is consistent with the OpenStack branch, "stable/mitakaocata" for example.
Code Block | ||||
---|---|---|---|---|
| ||||
$ git clone -b [stable/ocata or stable/mitakapike] https://git.openstack.org/openstack-dev/devstack |
The following is the example of local.conf. Please set the IP addresses correctly, and the network setting should be set properly as below.(Branches can be modified to stable/queens or stable/rocky for your wish)
Code Block | ||
---|---|---|
| ||
[[local|localrc]]
HOST_IP=10.134.231 | ||
Code Block | ||
| ||
[[local|localrc]] HOST_IP=10.134.231.28 SERVICE_HOST=10.134.231.28 RABBIT_HOST=10.134.231.28 DATABASE_HOST=10.134.231.28 Q_HOST=10.134.231.28 ADMIN_PASSWORD=nova DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD SERVICE_TOKEN=$ADMIN_PASSWORD DATABASE_TYPE=mysql # Log USE_SCREEN=True SCREEN_LOGDIR=/opt/stack/logs/screen LOGFILE=/opt/stack/logs/xstack.sh.log LOGDAYS=1 # ImagesForce config drive FORCE_CONFIG_DRIVE=True # Networks Q_ML2_TENANT_NETWORK_TYPE=vxlan Q_ML2_PLUGIN_MECHANISM_DRIVERS=onos_ml2 Q_PLUGIN_EXTRA_CONF_PATH=/opt/stack/networking-onos/etc/neutron/plugins/ml2 QML2_PLUGIN_EXTRATYPE_CONF_FILES=(ml2_conf_onos.ini)DRIVERS=flat,vlan,vxlan ML2_L3_PLUGIN=onos_router NEUTRON_CREATE_INITIAL_NETWORKS=False # Services enable_serviceplugin q-svc disable_service n-net disable_service n-cpu disable_service tempest disable_service c-sch disable_service c-api disable_service c-volnetworking-onos https://github.com/openstack/networking-onos.git stable/pike ONOS_MODE=controller_only # Services ENABLED_SERVICES=key,nova,n-api,n-cond,n-sch,n-novnc,n-cauth,placement-api,g-api,g-reg,q-svc,horizon,rabbit,mysql # Branches GLANCE_BRANCH=stable/mitakapike HORIZON_BRANCH=stable/mitakapike KEYSTONE_BRANCH=stable/mitakapike NEUTRON_BRANCH=stable/mitakapike NOVA_BRANCH=stable/mitakapike |
If you use other deployment tool or build OpenStack manually, refer to the following Nova and Neutron configurations.
Code Block | ||||
---|---|---|---|---|
| ||||
core_plugin = onos_ml2
service_plugins = onos_router
dhcp_agent_notification = False |
Code Block | ||
---|---|---|
| ||
[ml2] tenant_network_types = vxlan type_drivers = vxlanflat,vlan,vxlan mechanism_drivers = onos_ml2 [ml2_type_flat] flat_networks = public1, public2 #Whatever physical networks you want to create [securitygroup] enable_security_group = True |
...
Code Block | ||
---|---|---|
| ||
[DEFAULT] force_config_drive = True network_api_class = nova.network.neutronv2.api.API security_group_api = neutron [neutron] url = http://10.134.231.28:9696 auth_strategy = keystone admin_auth_url = http://10.134.231.28:35357/v2.0 admin_tenant_name = service admin_username = neutron admin_password = [admin passwd] |
Don't forget to set ml2_conf_onos.ini when you start Neutron service.
Code Block |
---|
/usr/bin/python /usr/local/bin/neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /opt/stack/networking-onos/etc/neutron/plugins/ml2/ml2_conf_onos.ini |
Compute node
...
Code Block | ||
---|---|---|
| ||
[[local|localrc]] HOST_IP=10.134.231.30 SERVICE_HOST=10.134.231.28 RABBIT_HOST=10.134.231.28 DATABASE_HOST=10.134.231.28 ADMIN_PASSWORD=nova DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD SERVICE_TOKEN=$ADMIN_PASSWORD DATABASE_TYPE=mysql NOVA_VNC_ENABLED=True VNCSERVER_PROXYCLIENT_ADDRESS=$HOST_IP VNCSERVER_LISTEN=$HOST_IP # Force config drive FORCE_CONFIG_DRIVE=True LIBVIRT_TYPE=kvm # should be qemu if your compute node is a vm # Log USE_SCREEN=True SCREEN_LOGDIR=/opt/stack/logs/screen LOGFILE=/opt/stack/logs/xstack.sh.log LOGDAYS=1 # Services ENABLED_SERVICES=n-cpu,placement-client,neutron enable_plugin networking-onos https://github.com/openstack/networking-onos.git stable/pike ONOS_MODE=compute # Branches NOVA_BRANCH=stable/mitakapike KEYSTONE_BRANCH=stable/mitakapike NEUTRON_BRANCH=stable/mitakapike |
Note |
---|
If your compute node is a VM, try http://docs.openstack.org/developer/devstack/guides/devstack-with-nested-kvm.html this first or set |
For manual set up, set Neutron as a network API in the Nova configuration.
Code Block | ||
---|---|---|
| ||
[DEFAULT] force_config_drive = True network_api_class = nova.network.neutronv2.api.API security_group_api = neutron [neutron] url = http://10.134.231.28:9696 auth_strategy = keystone admin_auth_url = http://10.134.231.28:35357/v2.0 admin_tenant_name = service admin_username = neutron admin_password = [admin passwd] |
Gateway node
No OpenStack service is required for a gateway node.
ONOS-SONA Setup
1. Refer to SONA Network Configuration Guide and write a network configuration file, typically named network-cfg.json. Place the configuration file under tools/package/config/, build, create package, and then install ONOS. Please note that it needs additional steps for building and installing openstackNetworking application due to the application does not fully support Buck build yet.
Code Block |
---|
# SONA cluster (1-node)
export OC1=onos-01
export ONOS_APPS="drivers,openflow-base" |
Code Block |
---|
onos$ buck build onos
onos$ cp ~/network-cfg.json ~/onos/tools/package/config/
onos$ onos-package
onos$ stc setup
onos$ onos-buck-publish-local
onos$ onos-buck publish --to-local-repo //protocols/ovsdb/api:onos-protocols-ovsdb-api
onos$ onos-buck publish --to-local-repo //protocols/ovsdb/rfc:onos-protocols-ovsdb-rfc
onos$ onos-buck publish --to-local-repo //apps/openstacknode:onos-apps-openstacknode
onos$ cd apps/openstacknetworking; mci;
onos$ onos-app $OC1 reinstall! target/onos-app-openstacknetworking-1.10.0-SNAPSHOT.oar |
2. Check all applications are activated successfully.
Code Block |
---|
onos> apps -a -s
* 9 org.onosproject.ovsdb-base 1.10.0.SNAPSHOT OVSDB Provider
* 13 org.onosproject.optical-model 1.10.0.SNAPSHOT Optical information model
* 20 org.onosproject.drivers 1.10.0.SNAPSHOT Default device drivers
* 39 org.onosproject.drivers.ovsdb 1.10.0.SNAPSHOT OVSDB Device Drivers
* 47 org.onosproject.openflow-base 1.10.0.SNAPSHOT OpenFlow Provider
* 56 org.onosproject.openstacknode 1.10.0.SNAPSHOT OpenStack Node Bootstrap App
* 57 org.onosproject.openstacknetworking 1.10.0.SNAPSHOT OpenStack Networking App |
3. Check all nodes are registered and all COMPUTE type node's states are COMPLETE with openstack-nodes command. Use openstack-node-check command for more detailed states if the state is INCOMPLETE. If you want to reinitialize only a particular compute node, use openstack-node-init command with hostname. For GATEWAY type node, leave it in DEVICE_CREATED state. You'll need additional configurations explained later for gateway nodes.
Code Block |
---|
onos> openstack-nodes
Hostname Type Integration Bridge Router Bridge Management IP Data IP VLAN Intf State
sona-compute-01 COMPUTE of:00000000000000a1 10.1.1.162 10.1.1.162 COMPLETE
sona-compute-02 COMPUTE of:00000000000000a2 10.1.1.163 10.1.1.163 COMPLETE
sona-gateway-02 GATEWAY of:00000000000000a4 of:00000000000000b4 10.1.1.165 10.1.1.165 DEVICE_CREATED
Total 3 nodes |
Gateway Node and ONOS-vRouter Setup
Single Gateway Node Setup
1. For all GATEWAY type nodes, Quagga and additional ONOS instance is required. Let's download and install Docker and required python packages first.
Code Block | ||
---|---|---|
| ||
$ wget -qO- https://get.docker.com/ | sudo sh
$ sudo apt-get install python-pip -y
$ sudo pip install oslo.config
$ sudo pip install ipaddress |
2. Download sona-setup scripts as well.
Code Block | ||
---|---|---|
| ||
$ git clone https://github.com/sonaproject/sona-setup.git
$ cd sona-setup |
3. Write vRouterConfig.ini and place it under sona-setup directory.
Code Block | ||||
---|---|---|---|---|
| ||||
[DEFAULT]
routerBridge = "of:00000000000000b1"
floatingCidr = "172.27.0.0/24,172.28.0.0/24"
floatingGateway = "172.27.0.1, 172.28.0.1/24"
localPeerMac = "fe:00:00:00:00:01"
localPeerIp = "172.18.0.254/30"
vRouterName = "gateway-01"
bgpNeighborIp = "172.18.0.253/30"
localAsNum = 65101
remoteAsNum = 65100
uplinkPortNum = "3" |
- line 2, routerBridge: Router bridge device ID configured in the network configuration. It should be unique across the system.
- line 3, floatingCidr: Floating IP address ranges. It can be comma separated list.
- line 4, floatingGateway: Gateway IP of Floating IP address range. It can be comma separated list.
- line 5, localPeerMac: Local MAC address used for peering. It should be unique across the system.
- line 6, localPeerIp: Local IP address used for peering. It should be unique across the system.
- line 7, vRouterName: Hostname to be configured in Quagga.
- line 8, bgpNeighborIp: Remote peer's IP address.
- line 9, localAsNum: Local AS number.
- line 10, remoteAsNum: Remote peer's AS number.
- line 11, uplinkPortNum: Port number of uplink interface on br-router bridge.
4. Run createJsonAndvRouter.sh. It will create configurations for vRouter, vrouter.json, and then brings up ONOS container with vRouter application activated.
Code Block |
---|
sona-setup$ ./createJsonAndvRouter.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e5ac67e62bbb onosproject/onos:1.6 "./bin/onos-service" 9 days ago Up 9 days 6653/tcp, 8101/tcp, 8181/tcp, 9876/tcp onos-vrouter |
5. Next, run createQuagga.sh. It will create Quagga configurations, zebra.conf and bgpd.conf with floating IP ranges, and then brings up Quagga container. It also re-generates vrouter.json with Quagga container's port number and restart ONOS container.
Code Block |
---|
sona-setup$ ./createQuagga.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
978dadf41240 onosproject/onos:1.6 "./bin/onos-service" 11 hours ago Up 11 hours 6653/tcp, 8101/tcp, 8181/tcp, 9876/tcp onos-vrouter
5bf4f2d59919 hyunsun/quagga-fpm "/usr/bin/supervisord" 11 hours ago Up 11 hours 179/tcp, 2601/tcp, 2605/tcp gateway-01 |
Code Block | ||
---|---|---|
| ||
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname gateway-01
password zebra
!
router bgp 65101
bgp router-id 172.18.0.254
timers bgp 3 9
neighbor 172.18.0.253 remote-as 65100
neighbor 172.18.0.253 ebgp-multihop
neighbor 172.18.0.253 timers connect 5
neighbor 172.18.0.253 advertisement-interval 5
network 172.27.0.0/24
!
log file /var/log/quagga/bgpd.log |
Code Block | ||
---|---|---|
| ||
!
hostname gateway-01
password zebra
!
fpm connection ip 172.17.0.2 port 2620 |
If you check the result of ovs-ofctl show
, there should be a new port named quagga
on br-router
bridge.
Code Block |
---|
sona-setup$ sudo ovs-ofctl show br-router
OFPT_FEATURES_REPLY (xid=0x2): dpid:00000000000000b1
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(patch-rout): addr:1a:46:69:5a:8e:f6
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(quagga): addr:7a:9b:05:57:2c:ff
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(br-router): addr:1a:13:72:57:4a:4d
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 |
6. If there's no external router or emulation of it in your setup, add another Quagga container which acts as an external router by running createQuaggaRouter.sh. It will create Quagga configurations for the external router and then brings up another Quagga container with them. It also fixes the uplink port number from the vRouterConfig.ini and vrouter.json with the newly added external router container's port number and then restarts ONOS.
Post Installation Setup
After installing OpenStack completely, please run the following configuration in the OpenStack controller node. It is required for OpenStack Ocata version, NOT because of SONA. Please replace OPENSTACK_CONTROLLER_HOST_IP to the correct IP address.
Code Block |
---|
$ nova-manage cell_v2 map_cell0 --database_connection 'mysql+pymysql://root:nova@OPENSTACK_CONTROLLER_HOST_IP/nova_cell0?charset=utf8'
$ nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:nova@OPENSTACK_CONTROLLER_HOST_IP:5672/
$ nova-manage cell_v2 discover_hosts |
Gateway node
No OpenStack service is required for a gateway node.
ONOS-SONA Setup
1. Refer to SONA Network Configuration Guide and write a network configuration file, typically named network-cfg.json. Place the configuration file under tools/package/config/, build, create package, and then install ONOS.
Note |
---|
Note that following tutorial is relevant only for building SONA apps against BUCK tool, and this only works from ONOS 1.13.0. So we strongly recommend the user to use ONOS 1.13.0 or above, if you would like to use BUCK to build SONA apps. For some reasons, if you have to use ONOS below 1.12.0, please use MAVEN to build the ONOS. |
Code Block |
---|
# SONA cluster (1-node)
export OC1=onos-01
export ONOS_APPS="drivers,openflow-base,openstacknetworking" |
In case you are using cell, here's example cell file for 3-node cluster
Code Block |
---|
export OC1=172.27.0.7
export OC2=172.27.0.8
export OC3=172.27.0.10
export ONOS_APPS="drivers,openflow-base,openstacknetworking" |
Code Block |
---|
onos$ ob
onos$ op
onos$ stc setup
onos$ curl --user onos:rocks -X POST -H "Content-Type: application/json" http://ONOS_IP:8181/onos/openstacknode/configure -d @network-cfg.json |
2. Check all applications are activated successfully.
Code Block |
---|
onos> apps -a -s
* 9 org.onosproject.ovsdb-base 1.13.0.SNAPSHOT OVSDB Provider
* 13 org.onosproject.optical-model 1.13.0.SNAPSHOT Optical information model
* 20 org.onosproject.drivers |
Code Block |
sona-setup$ ./createQuaggaRouter.sh sona-setup$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 978dadf41240 onosproject/onos:1.6 "./bin/onos-service" 11 hours ago Up 11 hours 6653/tcp, 8101/tcp, 8181/tcp, 9876/tcp onos-vrouter 32b10a038d78 hyunsun/quagga-fpm "/usr/bin/supervisord" 11 hours ago Up 11 hours 179/tcp, 2601/tcp, 2605/tcp router-01 5bf4f2d59919 hyunsun/quagga-fpm "/usr/bin/supervisord" 11 hours ago Up 11 hours 179/tcp, 2601/tcp, 2605/tcp gateway-01 |
Code Block | ||
---|---|---|
| ||
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname router-01
password zebra
!
router bgp 65100
bgp router-id 172.18.0.253
timers bgp 3 9
neighbor 172.18.0.254 remote-as 65101
neighbor 172.18.0.254 ebgp-multihop
neighbor 172.18.0.254 timers connect 5
neighbor 172.18.0.254 advertisement-interval 5
neighbor 172.18.0.254 default-originate
!
log file /var/log/quagga/bgpd.log |
Code Block | ||
---|---|---|
| ||
!
hostname router-01
password zebra
! |
If you check the result of ovs-ofctl show
, there should be a new port named quagga-router
on br-router
bridge.
Code Block | ||
---|---|---|
| ||
sona-setup$ sudo ovs-ofctl show br-routerOFPT_FEATURES_REPLY (xid=0x2): dpid:00000000000000b1
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(patch-rout): addr:1a:46:69:5a:8e:f6
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(quagga): addr:7a:9b:05:57:2c:ff
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
3(quagga-router): addr:c6:f5:68:d6:ff:56
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(br-router): addr:1a:13:72:57:4a:4d
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 |
7. Now, check hosts, fpm-connections, next-hops, and routes from ONOS-vRouter. You should be able to see default route (0.0.0.0/0) with next hop of the external router.
1.13.0.SNAPSHOT Default device drivers
* 39 org.onosproject.drivers.ovsdb 1.13.0.SNAPSHOT OVSDB Device Drivers
* 47 org.onosproject.openflow-base 1.13.0.SNAPSHOT OpenFlow Provider
* 56 org.onosproject.openstacknode 1.13.0.SNAPSHOT OpenStack Node Bootstrap App
* 57 org.onosproject.openstacknetworking 1.13.0.SNAPSHOT OpenStack Networking App |
3. Check all nodes are registered and all COMPUTE type node's states are COMPLETE with openstack-nodes command. Use openstack-node-check command for more detailed states if the state is INCOMPLETE. If you want to reinitialize only a particular compute node, use openstack-node-init command with hostname. If you have no physical peer switch, for GATEWAY type node, the state of the node would be DEVICE_CREATED state. You'll need additional configurations explained later for gateway nodes.
Code Block |
---|
onos> openstack-nodes
Hostname Type Integration Bridge Router Bridge Management IP Data IP VLAN Intf State
sona-compute-01 COMPUTE of:00000000000000a1 10.1.1.162 10.1.1.162 COMPLETE
sona-compute-02 COMPUTE of:00000000000000a2 |
Code Block |
onos> hosts id=FA:00:00:00:00:01/None, mac=FA:00:00:00:00:01, location=of:00000000000000b1/25, vlan=None, ip(s)=[172.18.0.253] id=FE:00:00:00:00:02/None, mac=FE:00:00:00:00:02, location=of:00000000000000b1/1, vlan=None, ip(s)=[172.27.0.1], name=FE:00:00:00:00:02/None onos> fpm-connections 172.17.0.3:52332 connected since 6m ago onos> next-hops ip=172.18.0.253, mac=FA:00:00:00:00:01, numRoutes=1 onos> routes Table: ipv4 Network Next Hop 0.0.0.0/0 17210.181.01.253 163 Total: 1 Table: ipv6 Network 10.1.1.163 Next Hop Total: 0 |
8. Add additional route for the floating IP ranges manually and check routes again.
Code Block |
---|
onos> route-add 172.27.0.0/24 172.27.0.1 onos> routes Table: ipv4 Network COMPLETE sona-gateway-02 GATEWAY Next Hop 0.0.0.0/0of:00000000000000a4 of:00000000000000b4 17210.181.01.253 165 172.27.0.0/24 17210.271.01.1 165 Total: 2 Table: ipv6 Network Next Hop Total: 0COMPLETE onos> next-hops ip=172.18.0.253, mac=FA:00:00:00:00:01, numRoutes=1 ip=Total 3 nodes |
Switch Setup
For switch to which Gatewy Node is connected, vlan and trunk setup is required.
1. Suppose we chose '172.27.0.1
...
/24' as floating IP range and Gateway Nodes are connected to switch via port 2,3. And you decided to assign vlan number 20 to floating IP range. In that case, switch setup should be like below(Arista syntax).
Code Block | ||
---|---|---|
| ||
Swtich(config)#interface vlan 20
Swtich(config-vlan-20)ip address 172.27.0.1/24
Swtich(config-vlan-20)no shutdown
Swtich(config)#interface ethernet 2-3
Swtich(config-if-Et1-2)#switchport mode trunk
Swtich(config-if-Et1-2)#switchport trunk allowed vlan 20
Swtich(config-if-Et1-2)#switchport trunk native vlan tag 20 |
2. If you need multiple floating IP ranges, for example 172.27.1.1/24 with vlan 200, additional setup is required(Arista syntax).
Code Block | ||
---|---|---|
| ||
Swtich(config)#interface vlan 20
Swtich(config-vlan-20)ip address 172.27.0.1/24
Swtich(config-vlan-20)no shutdown
Swtich(config)#interface vlan 200
Swtich(config-vlan-20)ip address 172.27.1.1/24
Swtich(config-vlan-20)no shutdown
Swtich(config)#interface ethernet 2-3
Swtich(config-if-Et1-2)#switchport mode trunk
Swtich(config-if-Et1-2)#switchport trunk allowed vlan 20-21
Swtich(config-if-Et1-2)#switchport trunk native vlan tag 20 |
Gateway Node Setup
Basically there's no additional setup is required on Gateway nodes. Those are for whom doen't have physical peer switch.
1. Let's download and install Docker first.
Code Block | ||
---|---|---|
| ||
$ wget -qO- https://get.docker.com/ | sudo sh |
2. Install and configure OVS
OVS version depends on the SONA features you want to enable. If you want staful NAT features, you have to install OVS 2.6 or higher. Otherwise, you can install OVS 2.5.
Then, set the OVSDB listener port as 6640 so that ONOS can intiated OVSDB connection.
Code Block |
---|
$ sudo ovs-vsctl set-manager ptcp:6640 |
Confgiure br-int bridge using openstack-node-init command.
Code Block |
---|
onos> openstack-node-init gateway-01
Initializing gateway-01
Done. |
You can check if br-int bridge is configured correctly using ovs-vsctl command, as follows.
Code Block |
---|
$ sudo ovs-vsctl show
427d7ee0-218f-4d68-b503-a5639a367357
Manager "ptcp:6640"
Bridge br-int
|
9. Everything's ready! Try init the gateway node again by running openstack-node-init command from ONOS-SONA.
Code Block |
---|
onos> openstack-node-init gateway-01 |
Code Block |
---|
onos> openstack-nodes Hostname Type Integration Bridge Router Bridge Management IP Data IP VLAN Intf State sona-compute-01 COMPUTE of:00000000000000a1 10.1.1.162 Controller "tcp:10.1.1.16230:6653" is_connected: true COMPLETE sona-compute-02fail_mode: secure COMPUTE Port br-int of:00000000000000a2 Interface br-int Port vxlan 10.1.1.163 Interface 10.1.1.163vxlan type: vxlan COMPLETE sona-gateway-02 GATEWAY options: {key=flow, remote_ip=flow} ofovs_version:00000000000000a4 of:00000000000000b4 10.1.1.165 10.1.1.165 COMPLETE Total 3 nodes |
Multiple Gateway Nodes Setup
SONA allows multiple gateway nodes for scalability as well as HA. Adding additional gatewy node is very easy. Just add the node configuration to ONOS-SONA network configuration and then try init to make the node state DEVICE_CREATED. And then do the same steps with the above single gateway node setup in the new gateway node. Don't forget to put unique value for quaggaMac and quaggaIp. Here is an example configuration of the second gateway node.
Code Block | ||
---|---|---|
| ||
[DEFAULT]
routerBridge = "of:00000000000000b2"
floatingCidr = "172.27.0.0/24"
dummyHostIp = "172.27.0.1"
quaggaMac = "fe:00:00:00:00:03"
quaggaIp = "172.18.0.250/30"
gatewayName = "gateway-02"
bgpNeighborIp = "172.18.0.249/30"
asNum = 65101
peerAsNum = 65100
uplinkPortNum = "3" |
You'll have to enable multipath in your external router as well.
Code Block |
---|
router bgp 65100
timers bgp 3 9
distance bgp 20 200 200
maximum-paths 2 ecmp 2
neighbor 172.18.0.254 remote-as 65101
neighbor 172.18.0.254 maximum-routes 12000
neighbor 172.18.0.250 remote-as 65101
neighbor 172.18.0.250 maximum-routes 12000
redistribute connected |
"2.5.2" |
3. Download sona-setup scripts as well.
Code Block | ||
---|---|---|
| ||
$ git clone https://github.com/sonaproject/sona-setup.git
$ cd sona-setup |
4. Write externalRouterConfig.ini and place it under sona-setup directory.
Code Block | ||||
---|---|---|---|---|
| ||||
floatingCidr = "172.27.0.1/24"
externalPeerMac = "fa:00:00:00:00:01" |
- line 1, floatingCidr: Floating IP address ranges. It can be comma separated list.
- line 2, externalPeerMac: Remote peer router's MAC address.
5. Run createExternalRouter.sh. It will create emulated external peer router.
Code Block |
---|
sona-setup$ ./createExternalRouter.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5885654827e2 opensona/docker-quagga "/usr/bin/supervisord" 3 weeks ago Up 3 weeks 179/tcp, 2601/tcp, 2605/tcp router |
*Note that we don't use quagga app. We just use quagga container for convenience.
6. When every work is done, you create router with appropriate extenal network that floating IP range is assigned. Then you execute below CLI to check MAC learning for external peer rotuer is working well,
Code Block |
---|
onos> openstack-peer-routers
Router IP Mac Address VLAN ID
172.27.0.1 FA:00:00:00:00:01 None |
Code Block |
#routed port connected to gateway-01
interface Ethernet43
no switchport
ip address 172.18.0.253/30
#routed port connected to gateway-02
interface Ethernet44
no switchport
ip address 172.18.0.249/30 |
HA Setup
Basically, ONOS itself provides HA by default when there are multiple instances in the cluster. This section describes how to add a proxy server beyond the ONOS cluster, and make use of it in Neutron as a single access point of the cluster. For the proxy server, we used the HA proxy server (http://www.haproxy.org) here.
1. Install HA proxy.
Code Block | ||
---|---|---|
| ||
$ sudo add-apt-repository -y ppa:vbernat/haproxy-1.5 $ sudo add-apt-repository -y ppa:vbernat/haproxy-1.5 $ sudo apt-get update $ sudo apt-get install -y haproxy |
2. Configure HA proxy.
Code Block | ||
---|---|---|
| ||
global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http frontend localnodes bind *:8181 mode http default_backend nodes backend nodes mode http balance roundrobin option forwardfor http-request set-header X-Forwarded-Port %[dst_port] http-request add-header X-Forwarded-Proto https if { ssl_fc } option httpchk GET /onos/ui/login.html server web01 [onos-01 IP address]:8181 check server web02 [onos-02 IP address]:8181 check server web03 [onos-03 IP address]:8181 check listen stats *:1936 stats enable stats uri / stats hide-version stats auth someuser:password |
3. Set url_path to point to the proxy server in Neutron ML2 ONOS mechanism driver configuration and restart Neutron.
Code Block | ||
---|---|---|
| ||
# Configuration options for ONOS ML2 Mechanism driver [onos] # (StrOpt) ONOS ReST interface URL. This is a mandatory field. url_path = http://[proxy-server IP]:8181/onos/openstackswitching # (StrOpt) Username for authentication. This is a mandatory field. username = onos # (StrOpt) Password for authentication. This is a mandatory field. password = rocks |
...
4. Stop one of the ONOS instance and check everything works fine.
...