...
| Code Block |
|---|
onos> openstack-nodes
Hostname Type Integration Bridge Router Bridge Management IP Data IP VLAN Intf State
sona-compute-01 COMPUTE of:00000000000000a1 10.1.1.162 10.1.1.162 COMPLETE
sona-compute-02 COMPUTE of:00000000000000a2 10.1.1.163 10.1.1.163 COMPLETE
sona-gateway-02 GATEWAY of:00000000000000a4 of:00000000000000b4 10.1.1.165 10.1.1.165 DEVICE_CREATED
Total 3 nodes |
...
Switch Setup
Single Gateway Node Setup
1. For all GATEWAY type nodes, Quagga and additional ONOS instance is required. Let's download and install Docker and required python packages first.
| Code Block |
|---|
|
$ wget -qO- https://get.docker.com/ | sudo sh
$ sudo apt-get install python-pip -y
$ sudo pip install oslo.config
$ sudo pip install ipaddress |
For switch to which Gatewy Node is connected, vlan and trunk setup is required.
1. Suppose we chose '172.27.0.1/24' as floating IP range and Gateway Nodes are connected to switch via port 2,3. And you decided to assign vlan number 20 to floating IP range. In that case, switch setup should be like below(Arista syntax)2. Download sona-setup scripts as well.
| Code Block |
|---|
|
$ git clone https://github.com/sonaproject/sona-setup.git
$ cd sona-setup |
3. Write vRouterConfig.ini and place it under sona-setup directory.
| Code Block |
|---|
| title | vRouterConfig.ini |
|---|
| linenumbers | true |
|---|
|
[DEFAULT]
routerBridge = "of:00000000000000b1"
floatingCidr = "Swtich(config)#interface vlan 20
Swtich(config-vlan-20)ip address 172.27.0.0/24,172.28.0.0/24"
localPeerMac = "fe:00:00:00:00:01"
localPeerIp = "172.18.0.254/30"
vRouterName = "gateway-01"
bgpNeighborIp = "172.18.0.253/30"
localAsNum = 65101
remoteAsNum = 65100
uplinkPortNum = "3" |
- line 2, routerBridge: Router bridge device ID configured in the network configuration. It should be unique across the system.
- line 3, floatingCidr: Floating IP address ranges. It can be comma separated list.
- line 4, localPeerMac: Local MAC address used for peering. It should be unique across the system.
- line 5, localPeerIp: Local IP address used for peering. It should be unique across the system.
- line 6, vRouterName: Hostname to be configured in Quagga.
- line 7, bgpNeighborIp: Remote peer's IP address.
- line 8, localAsNum: Local AS number.
- line 9, remoteAsNum: Remote peer's AS number.
- line 10, uplinkPortNum: Port number of uplink interface on br-router bridge.
4. Run createJsonAndvRouter.sh. It will create configurations for vRouter, vrouter.json, and then brings up ONOS container with vRouter application activated.
| Code Block |
|---|
sona-setup$ ./createJsonAndvRouter.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e5ac67e62bbb onosproject/onos:1.6 "./bin/onos-service" 9 days ago Up 9 days 6653/tcp, 8101/tcp, 8181/tcp, 9876/tcp onos-vrouter |
5. Next, run createQuagga.sh. It will create Quagga configurations, zebra.conf and bgpd.conf with floating IP ranges, and then brings up Quagga container. It also re-generates vrouter.json with Quagga container's port number and restart ONOS container.
| Code Block |
|---|
sona-setup$ ./createQuagga.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
978dadf41240 onosproject/onos:1.6 "./bin/onos-service" 11 hours ago Up 11 hours 6653/tcp, 8101/tcp, 8181/tcp, 9876/tcp onos-vrouter
5bf4f2d59919 hyunsun/quagga-fpm "/usr/bin/supervisord" 11 hours ago Up 11 hours 179/tcp, 2601/tcp, 2605/tcp gateway-01 |
| Code Block |
|---|
| title | volumes/gateway/bgpd.conf |
|---|
|
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname gateway-01
password zebra
!
router bgp 65101
bgp router-id 172.18.0.254
timers bgp 3 9
neighbor 172.18.0.253 remote-as 65100
neighbor 172.18.0.253 ebgp-multihop
neighbor 172.18.0.253 timers connect 5
neighbor 172.18.0.253 advertisement-interval 5
network 172.27.0.0/24
!
log file /var/log/quagga/bgpd.log |
| Code Block |
|---|
| title | volumes/gateway/zebra.conf |
|---|
|
!
hostname gateway-01
password zebra
!
fpm connection ip 172.17.0.2 port 2620 |
...
| Code Block |
|---|
sona-setup$ sudo ovs-ofctl show br-router
OFPT_FEATURES_REPLY (xid=0x2): dpid:00000000000000b1
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(patch-rout): addr:1a:46:69:5a:8e:f6
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(quagga): addr:7a:9b:05:57:2c:ff
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(br-router): addr:1a:13:72:57:4a:4d
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 |
Next, set the controller of the br-router to the ONOS controller container executed by the script.
| Code Block |
|---|
$ sudo ovs-vsctl set-controller br-router tcp:172.17.0.2:6653 |
Pleaset note that each gateway has its own ONOS controller, which is different from the main ONOS controller. In most of the case, the ONOS container is executed with the IP address of 172.17.0.2. However, if the IP is taken for some reasons, then another IP address can be assigned. So, please check the IP address after running the vRouter script, as below.
| Code Block |
|---|
Finished setup ONOS-vRouter!
Access ONOS-vRouter with 'ssh -p 8101 karaf@172.17.0.2' password 'karaf' |
...
1/24
Swtich(config-vlan-20)no shutdown
Swtich(config)#interface ethernet 2-3
Swtich(config-if-Et1-2)#switchport mode trunk
Swtich(config-if-Et1-2)#switchport trunk allowed vlan 20
Swtich(config-if-Et1-2)#switchport trunk native vlan tag 20 |
2. If you need multiple floating IP ranges, for example 172.27.1.1/24 with vlan 200, additional setup is required(Arista syntax).
| Code Block |
|---|
|
Swtich(config)#interface vlan 20
Swtich(config-vlan-20)ip address 172.27.0.1/24
Swtich(config-vlan-20)no shutdown
Swtich(config)#interface vlan 200
Swtich(config-vlan-20)ip address 172.27.1.1/24
Swtich(config-vlan-20)no shutdown
Swtich(config)#interface ethernet 2-3
Swtich(config-if-Et1-2)#switchport mode trunk
Swtich(config-if-Et1-2)#switchport trunk allowed vlan 20-21
Swtich(config-if-Et1-2)#switchport trunk native vlan tag 20 |
Gateway Node Setup
Basically there's no additional setup is required on Gateway nodes. Those are for whom doen't have physical peer switch.
1. Let's download and install Docker first.
| Code Block |
|---|
|
$ wget -qO- https://get.docker.com/ | sudo sh |
2. Download sona-setup scripts as well.
| Code Block |
|---|
|
$ git clone -b 1.13 https://github.com/sonaproject/sona-setup.git
$ cd sona-setup |
3. Write externalRouterConfig.ini and place it under sona-setup directory.
| Code Block |
|---|
| title | externalRouterConfig.ini |
|---|
| linenumbers | true |
|---|
|
floatingCidr = "172.27.0.1/24"
externalPeerMac = "fa:00:00:00:00:01" |
- line 1, floatingCidr: Floating IP address ranges. It can be comma separated list.
- line 2, externalPeerMac: Remote peer router's MAC address.
4. Run createExternalRouter.sh. It will create emulated external peer router.
| Code Block |
|---|
sona-setup$ ./createExternalRouter.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE |
| Code Block |
|---|
sona-setup$ ./createQuaggaRouter.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
978dadf41240 onosproject/onos:1.6 "./bin/onos-service" 11 hours ago Up 11 hours 6653/tcp, 8101/tcp, 8181/tcp, 9876/tcp onos-vrouter
32b10a038d78 hyunsun/quagga-fpm "/usr/bin/supervisord" 11 hours ago Up 11 hours 179/tcp, 2601/tcp, 2605/tcp router-01
5bf4f2d59919 hyunsun/quagga-fpm "/usr/bin/supervisord" 11 hours ago Up 11 hours 179/tcp, 2601/tcp, 2605/tcp gateway-01 |
| Code Block |
|---|
| title | volumes/router/bgpd.conf |
|---|
|
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname router-01
password zebra
!
router bgp 65100
bgp router-id 172.18.0.253
timers bgp 3 9
neighbor 172.18.0.254 remote-as 65101
neighbor 172.18.0.254 ebgp-multihop
neighbor 172.18.0.254 timers connect 5
neighbor 172.18.0.254 advertisement-interval 5
neighbor 172.18.0.254 default-originate
!
log file /var/log/quagga/bgpd.log |
| Code Block |
|---|
| title | volumes/router/zebra.conf |
|---|
|
!
hostname router-01
password zebra
! |
If you check the result of ovs-ofctl show, there should be a new port named quagga-router on br-router bridge.
| Code Block |
|---|
| title | volumes/router/zebra.conf |
|---|
|
sona-setup$ sudo ovs-ofctl show br-router
OFPT_FEATURES_REPLY (xid=0x2): dpid:00000000000000b1
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(patch-rout): addr:1a:46:69:5a:8e:f6
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(quagga): addr:7a:9b:05:57:2c:ff
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
3(quagga-router): addr:c6:f5:68:d6:ff:56
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(br-router): addr:1a:13:72:57:4a:4d
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 |
7. Now, check hosts, fpm-connections, next-hops, and routes from ONOS-vRouter. You should be able to see default route (0.0.0.0/0) with next hop of the external router.
| Code Block |
|---|
onos> hosts
id=FA:00:00:00:00:01/None, mac=FA:00:00:00:00:01, location=of:00000000000000b1/25, vlan=None, ip(s)=[172.18.0.253]
id=FE:00:00:00:00:02/None, mac=FE:00:00:00:00:02, location=of:00000000000000b1/1, vlan=None, ip(s)=[172.27.0.1], name=FE:00:00:00:00:02/None
onos> fpm-connections
172.17.0.3:52332 connected since 6m ago
onos> next-hops
ip=172.18.0.253, mac=FA:00:00:00:00:01, numRoutes=1
onos> routes
Table: ipv4
Network Next Hop
0.0.0.0/0 172.18.0.253
Total: 1
Table: ipv6
Network Next Hop
Total: 0 |
8. Add additional route for the floating IP ranges manually and check routes again.
| Code Block |
|---|
onos> route-add 172.27.0.0/24 172.27.0.1
onos> routes
Table: ipv4
Network Next Hop
0.0.0.0/0 172.18.0.253
172.27.0.0/24 172.27.0.1
Total: 2
Table: ipv6
Network Next Hop
Total: 0
onos> next-hops
ip=172.18.0.253, mac=FA:00:00:00:00:01, numRoutes=1
ip=172.27.0.1, mac=FE:00:00:00:00:02, numRoutes=1 |
9. Everything's ready! Try init the gateway node again by running openstack-node-init command from ONOS-SONA.
| Code Block |
|---|
onos> openstack-node-init gateway-01 |
| Code Block |
|---|
onos> openstack-nodes
Hostname Type Integration Bridge Router Bridge Management IP Data IP VLAN Intf State
sona-compute-01 COMPUTE of:00000000000000a1 10.1.1.162 10.1.1.162 COMPLETE
sona-compute-02 COMPUTE of:00000000000000a2 10.1.1.163 10.1.1.163 COMPLETE
sona-gateway-02 GATEWAY of:00000000000000a4 of:00000000000000b4 10.1.1.165 10.1.1.165 COMPLETE
Total 3 nodes |
Multiple Gateway Nodes Setup
SONA allows multiple gateway nodes for scalability as well as HA. Adding additional gatewy node is very easy. Just add the node configuration to ONOS-SONA network configuration and then try init to make the node state DEVICE_CREATED. And then do the same steps with the above single gateway node setup in the new gateway node. Don't forget to put unique value for quaggaMac and quaggaIp. Here is an example configuration of the second gateway node.
| Code Block |
|---|
|
[DEFAULT]
routerBridge = "of:00000000000000b2"
floatingCidr = "172.27.0.0/24"
floatingGateway = "172.27.0.1"
localPeerMac = "fe:00:00:00:00:03"
localPeerIp = "172.18.0.250/30"
vRouterName = "gateway-02"
bgpNeighborIp = "172.18.0.249/30"
localAsNum = 65101
remoteAsNum = 65100
uplinkPortNum = "3" |
You'll have to enable multipath in your external router as well.
| Code Block |
|---|
router bgp 65100
timers bgp 3 9
distance bgp 20 200 200
maximum-paths 2 ecmp 2
neighbor 172.18.0.254 remote-as 65101
neighbor 172.18.0.254 maximum-routes 12000
neighbor 172.18.0.250 remote-as 65101
neighbor 172.18.0.250 maximum-routes 12000
redistribute connected |
COMMAND CREATED STATUS PORTS NAMES
5885654827e2 opensona/docker-quagga "/usr/bin/supervisord" 3 weeks ago Up 3 weeks 179/tcp, 2601/tcp, 2605/tcp router |
*Note that we don't use quagga app. We just use quagga container for convenience.
5. When every work is done, you create router with appropriate extenal network that floating IP range is assigned. Then you execute below CLI to check MAC learning for external peer rotuer is working well,
| Code Block |
|---|
onos> openstack-peer-routers
Router IP Mac Address VLAN ID
172.27.0.1 FA:00:00:00:00:01 None |
| Code Block |
|---|
#routed port connected to gateway-01
interface Ethernet43
no switchport
ip address 172.18.0.253/30
#routed port connected to gateway-02
interface Ethernet44
no switchport
ip address 172.18.0.249/30 |
HA Setup
Basically, ONOS itself provides HA by default when there are multiple instances in the cluster. This section describes how to add a proxy server beyond the ONOS cluster, and make use of it in Neutron as a single access point of the cluster. For the proxy server, we used the HA proxy server (http://www.haproxy.org) here.
...