Versions Compared

Old Version 93

changes.mady.by.user Daniel Park

Saved on

compared with

New Version 94

changes.mady.by.user Hyunsun Moon

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Improved multiple gateway nodes setup

...

Node and ONOS-vRouter Setup

Single Gateway Node Setup

1. Push network-cfg.json after ONOS-SONA and OpenStack are ready, and check all COMPUTE type nodestate is COMPLETE with openstack-nodes command. Use openstack-node-check command for more detailed states of the node if it's INCOMPLETE. For GATEWAY type node, leave it in DEVICE_CREATED state. You'll need additional configurations explained later. 

...

If you check the result of ovs-vsctl show, there should be a new port named quagga on br-router bridge. 

 

6. If you have more gateways, you should differentiate router-id and MAC address of the gateway from others. Here's an example configuration for 2 gateways, If there's no external router or emulation of it in your setup, add another Quagga container which acts as an external router. First, modify volumes/router/zebra.conf and volumes/router/bgpd.conf to make this Quagga an external router neighboring with the one created right before, and use the same command above but with additional argument --external-router option to bring up the router container. You can put any MAC address this time.

Code Block
titlevolumes/router/bgpd.conf
! -*- bgp -*-
Code Block
titlevolumes/gateway/bgpd.conf
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname gatewayrouter-0201
password zebra
!
router bgp 6510165100
  bgp router-id 172.18.0.2501
  timers bgp 3 9
  neighbor 172.18.0.1254 remote-as 6510065101
  neighbor 172.18.0.1254 ebgp-multihop
  neighbor 172.18.0.1254 timers connect 5
  neighbor 172.18.0.1254 advertisement-interval 5
  networkneighbor 172.2718.0.0/24254 default-originate
!
log file /var/log/quagga/bgpd.loglog 
Code Block
titlevolumes/gatewayrouter/zebra.conf
!
hostname gatewayrouter-0201
password zebra
!
fpm connection ip 172.17.0.2 port 2620
Code Block
languagebash
$ ./quagga.sh --name=gatewayrouter-0201 --ip=172.18.0.2501/30 --mac=fefa:00:00:00:00:03
Code Block
titlephysical router interfaces
#routed port connected to gateway-01
interface Ethernet43
   no switchport
   ip address 172.18.0.253/30
#routed port connected to gateway-02
interface Ethernet44
   no switchport
   ip address 172.18.0.249/30
 
Code Block
titlebgp configuration on the physical router
router bgp 65100
   timers bgp 3 9
   distance bgp 20 200 200
   maximum-paths 2 ecmp 2
   neighbor 172.18.0.254 remote-as 65101
   neighbor 172.18.0.254 maximum-routes 12000
   neighbor 172.18.0.250 remote-as 65101
   neighbor 172.18.0.250 maximum-routes 12000
   ............

 

7. If there's no external router or emulation of it in your setup, add another Quagga container which acts as an external router. First, modify volumes/router/zebra.conf and volumes/router/bgpd.conf to make this Quagga an external router neighboring with the one created right before, and use the same command above but with additional argument --external-router option to bring up the router container. You can put any MAC address this time.

Code Block
titlevolumes/router/bgpd.conf
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname router-01
password zebra
!
router bgp 65100
  bgp router-id 172.18.0.1
  timers bgp 3 9
  neighbor 172.18.0.254 remote-as 65101
  neighbor 172.18.0.254 ebgp-multihop
  neighbor 172.18.0.254 timers connect 5
  neighbor 172.18.0.254 advertisement-interval 5
  neighbor 172.18.0.254 default-originate
!
log file /var/log/quagga/bgpd.log 
Code Block
titlevolumes/router/zebra.conf
!
hostname router-01
password zebra
!
Code Block
languagebash
$ ./quagga.sh --name=router-01 --ip=172.18.0.1/30 --mac=fa:00:00:00:00:01 --external-router

If you check the result of ovs-vsctl show, there should be a new port named quagga-router on br-router bridge. 

 

8. Once all the containers are up and running, check ports result from the ONOS-vRouter. If any port number does not match to the ones in vrouter.json, modify the config file with the correct port numbers, and just re-run the vrouter.sh. It actually happens often since you may re-create Quagga containers multiple times to fix the Quagga config files or for some other reasons, and OVS increases port number whenever a new port is added to a bridge. 

Code Block
$ ssh -p 8101 karaf@172.17.0.2
# password is karaf

onos> ports
id=of:00000000000000b1, available=true, role=MASTER, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.5.0, serial=None, driver=softrouter, channelId=172.17.0.1:58292, managementAddress=172.17.0.1, name=of:00000000000000b1, protocol=OF_13
  port=local, state=disabled, type=copper, speed=0 , portName=br-router, portMac=e6:a0:79:f9:d1:4a
  port=1, state=enabled, type=copper, speed=0 , portName=patch-rout, portMac=fe:da:85:15:b1:bf
  port=24, state=enabled, type=copper, speed=10000 , portName=quagga, portMac=06:96:1b:36:32:77
  port=25, state=enabled, type=copper, speed=10000 , portName=quagga-router, portMac=ea:1e:71:d1:fd:81
01 --external-router

If you check the result of ovs-vsctl show, there should be a new port named quagga-router on br-router bridge. 

 

7. Once all the containers are up and running, check ports result from the ONOS-vRouter. If any port number does not match to the ones in vrouter.json, modify the config file with the correct port numbers, and just re-run the vrouter.sh. It actually happens often since you may re-create Quagga containers multiple times to fix the Quagga config files or for some other reasons, and OVS increases port number whenever a new port is added to a bridge. 

Code Block
$ ssh -p 8101 karaf@172.17.0.2
# password is karaf

onos> ports
id=of:00000000000000b1, available=true, role=MASTER, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.5.0, serial=None, driver=softrouter, channelId=172.17.0.1:58292, managementAddress=172.17.0.1, name=of:00000000000000b1, protocol=OF_13
  port=local, state=disabled, type=copper, speed=0 , portName=br-router, portMac=e6:a0:79:f9:d1:4a
  port=1, state=enabled, type=copper, speed=0 , portName=patch-rout, portMac=fe:da:85:15:b1:bf
  port=24, state=enabled, type=copper, speed=10000 , portName=quagga, portMac=06:96:1b:36:32:77
  port=25, state=enabled, type=copper, speed=10000 , portName=quagga-router, portMac=ea:1e:71:d1:fd:81
Code Block
linenumberstrue
    "apps" : {
        "org.onosproject.router" : {
            "router" : {
                "controlPlaneConnectPoint" : "of:00000000000000b1/24",
                "ospfEnabled" : "true",
                "interfaces" : [ "b1-1", "b1-2" ]
            }
        }
    },
    "ports" : {
        "of:00000000000000b1/25" : {
            "interfaces" : [
                {
                    "name" : "b1-1",
                    "ips"  : [ "172.18.0.254/24" ],
Code Block
linenumberstrue
    "apps" : {
        "org.onosproject.router" : {
            "router" : {
                "controlPlaneConnectPointmac"  : "of:00000000000000b1/24",
fe:00:00:00:00:01"
                }
            ]
      "ospfEnabled" : "true"},
        "of:00000000000000b1/1" : {
            "interfaces" : [
                {
                   "b1-1", "b1-2" ]
            }
        }
    },
    "ports" : {
        "of:00000000000000b1/25" : {
            "interfaces" : [
                {
                    "name" : "b1-1",
                    "ips"  : [ "172.18.0.254/24" ],
                    "mac"  : "fe "name" : "b1-2",
                    "ips"  : [ "172.27.0.254/24" ],
                    "mac"  : "fe:00:00:00:00:01"
                }
            ]
        }
    },
    "hosts" : {
        "fe:00:00:00:00:02/-1" : {
            "basic": {
                "ips": ["172.27.0.1"],
                "location": "of:00000000000000b1/1"
            }
        }
    }
  • Line 4: Device ID and port number of the port with portName=quagga -> controlPlaneConnectPoint
  • Line 11: Device ID and port number of the port with portName=quagga-router or other actual uplink port.

If you have a floating range, 172.27.0.0/24 in this example, check the following configurations also.

  • Line 20: (optional interface config for floating IP address range) Device ID and port number of the port with portName=patch-rout
  • Line 34: (optional interface config for floating IP gateway) Device ID and port number of the port with portName=patch-rout 

 

Once you fix the vrouter.json file, re-create onos-vrouter container with the updated configuration. vrouter.sh script will take care of removing the existing container.

Code Block
sona-setup$ vrouter.sh

 

8. If everything's right, check fpm-connections, hosts and routes. 172.18.0.1 is the external default gateway in this example. If you added interface and host for floating IP range, you should be able to see the host in the list.

Code Block
onos> hosts
id=FA:00:00:00:00:01/None, mac=FA:00:00:00:00:01, location=of:00000000000000b1/25, vlan=None, ip(s)=[172.18.0.1]
id=FE:00:00:00:00:01"
                }
            ]
        },
        "of:00000000000000b1/1" : {
            "interfaces" : [
                {
                    "name" : "b1-2",
                    "ips"  : [ "172.27.0.254/24" ],
                    "mac"  : "fe/None, mac=FE:00:00:00:00:01, location=of:00000000000000b1/24, vlan=None, ip(s)=[172.18.0.254]
id=FE:00:00:00:00:02/None, mac=FE:00:00:00:00:02, location=of:00000000000000b1/1, vlan=None, ip(s)=[172.27.0.1], name=FE:00:00:00:00:02/None

onos> fpm-connections
172.17.0.2:52332 connected since 6m ago

onos> next-hops
ip=172.18.0.1, mac=FA:00:00:00:00:01"
                }, numRoutes=1

onos> routes
Table: ipv4
   Network         ]
   Next     }Hop
    },
0.0.0.0/0    "hosts" : {
    172.18.0.1
    "fe:00:00:00:00:02/-1" : {Total: 1

Table: ipv6
   Network         "basic": {
  Next Hop
   Total: 0


9. Add route for floating IP range manually and check the route is added.

Code Block
onos>         "ips": ["route-add 172.27.0.0/24 172.27.0.1"],
                "location": "of:00000000000000b1/1"onos> routes
Table: ipv4
    Network        }
    Next    }
    }

Line 4: Device ID and port number of the port with portName=quagga -> controlPlaneConnectPoint

Line 11: Device ID and port number of the port with portName=quagga-router or other actual uplink port

If you have a floating range, 172.27.0.0/24 in this example, check the following configurations also.

Line 20: (optional interface config for floating IP address range) Device ID and port number of the port with portName=patch-rout

Line 34: (optional interface config for floating IP gateway) Device ID and port number of the port with portName=patch-rout 

 

Once you fix the vrouter.json file, re-create onos-vrouter container with the updated configuration. vrouter.sh script will take care of removing the existing container.

Code Block
sona-setup$ vrouter.sh

 

Hop
   0.0.0.0/0          172.18.0.1
   172.27.0.0/24      172.27.0.1
   Total: 2

Table: ipv6
   Network            Next Hop
   Total: 0

onos> next-hops
ip=172.18.0.1, mac=FA:00:00:00:00:01, numRoutes=1
ip=172.27.0.1, mac=FE:00:00:00:00:02, numRoutes=1


10. Now 9. If everything's right, check fpm-connections, hosts and routes. 172.18.0.1 is the external default gateway in this example. If you added interface and host for floating IP range, you should be able to see the host in the listgateway node is in COMPLETE state when you re-trigger node initialization. You can either run command openstack-node-init gateway-01 or push the network configuration file again.

Code Block
onos> hosts
id=FA:00:00:00:00:01/None, mac=FA:00:00:00:00:01, location=of:00000000000000b1/25, vlan=None, ip(s)=[172.18.0.1]
id=FE:00:00:00:00:01/None, mac=FE:00:00:00:00:01, location=of:00000000000000b1/24, vlan=None, ip(s)=[172.18.0.254]
id=FE:00:00:00:00:02/None, mac=FE:00:00:00:00:02, location=of:00000000000000b1/1, vlan=None, ip(s)=[172.27.0.1], name=FE:00:00:00:00:02/None

onos> fpm-connections
172.17.0.2:52332 connected since 6m ago

onos> next-hops
ip=172.18.0.1, mac=FA:00:00:00:00:01, numRoutes=1

onos> routes
Table: ipv4
   Network            Next Hop
   0.0.0.0/0          172.18.0.1
   Total: 1

Table: ipv6
   Network            Next Hop
   Total: 0

10. Add route for floating IP range manually and check the route is added.

Code Block
onos> route-add 172.27.0.0/24 172.27.0.1
onos> routes
Table: ipv4
   Network            Next Hop
   0.0.0.0/0          172.18.0.1
   172.27.0.0/24      172.27.0.1
   Total: 2

Table: ipv6
   Network            Next Hop
   Total: 0

onos> next-hops
ip=172.18.0.1, mac=FA:00:00:00:00:01, numRoutes=1
ip=172.27.0.1, mac=FE:00:00:00:00:02, numRoutes=1

11. Now you should be able to see the gateway node is in COMPLETE state when you re-trigger node initialization. You can either run command openstack-node-init gateway-01 or push the network configuration file again.

Code Block
onos> openstack-node-init gateway-01
onos> openstack-node-init gateway-02

Or

Code Block
titlehow to push network config
$ curl --user onos:rocks -X POST -H "Content-Type: application/json" http://10.134.231.29:8181/onos/v1/network/configuration/ -d @network-cfg.json

...

 openstack-node-init gateway-01
onos> openstack-node-init gateway-02

Or

Code Block
titlehow to push network config
$ curl --user onos:rocks -X POST -H "Content-Type: application/json" http://10.134.231.29:8181/onos/v1/network/configuration/ -d @network-cfg.json


Code Block
onos> openstack-nodes
hostname=compute-01, type=COMPUTE, managementIp=10.134.231.30, dataIp=10.134.34.222, intBridge=of:00000000000000a1, routerBridge=Optional.empty init=COMPLETE
hostname=compute-02, type=COMPUTE, managementIp=10.134.231.31, dataIp=10.134.34.223, intBridge=of:00000000000000a2, routerBridge=Optional.empty init=COMPLETE
hostname=gateway-01, type=GATEWAY, managementIp=10.134.231.32, dataIp=10.134.33.224, intBridge=of:00000000000000a3, routerBridge=Optional[of:00000000000000b1] init=COMPLETE
hostname=gateway-02, type=GATEWAY, managementIp=10.134.231.33, dataIp=10.134.33.225, intBridge=of:00000000000000a4, routerBridge=Optional[of:00000000000000b2] init=COMPLETE
Total 4 nodes

Multiple Gateway Nodes Setup

SONA allows multiple gateway nodes for HA as well as scalability. Here's another example of multiple gateway nodes and external upstream router. In this scenario, each gateway node should have unique IP and MAC address for peering so that the upstream router can handle each of them as a different router. ONOS scalable gateway application is responsible for taking upstream packet through one of the gateway nodes, and the upstream router is responsible for taking downstream packet through one of the gateway nodes.

 

The following is an example Quagga configuration of the second gateway node.

 

Code Block
titlevolumes/gateway/bgpd.conf
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname gateway-02
password zebra
!
router bgp 65101
  bgp router-id 172.18.0.249
  timers bgp 3 9
  neighbor 172.18.0.1 remote-as 65100
  neighbor 172.18.0.1 ebgp-multihop
  neighbor 172.18.0.1 timers connect 5
  neighbor 172.18.0.1 advertisement-interval 5
  network 172.27.0.0/24
!
log file /var/log/quagga/bgpd.log

 

 

 

Code Block
titlevolumes/gateway/zebra.conf
!
hostname gateway-02
password zebra
!
fpm connection ip 172.17.0.2 port 2620

 

 

 

Once you are done with the configurations for the second Quaaga, run quagga.sh script to bring up Quagga container with the MAC address different from the one used for the first gateway node Quagga container.

 

Code Block
languagebash
$ ./quagga.sh --name=gateway-02 --ip=172.18.0.250/30 --mac=fe:00:00:00:00:03

 

 

 

Now configure the upstream router as below.

 

Code Block
titlebgp configuration on the physical router
router bgp 65100
   timers bgp 3 9
   distance bgp 20 200 200
   maximum-paths 2 ecmp 2
   neighbor 172.18.0.254 remote-as 65101
   neighbor 172.18.0.254 maximum-routes 12000
   neighbor 172.18.0.249 remote-as 65101
   neighbor 172.18.0.249 maximum-routes 12000
   redistribute connected

 

 

 

Code Block
titlephysical router interfaces
#routed port connected to gateway-01
interface Ethernet43
   no switchport
   ip address 172.18.0.253/30
#routed port connected to gateway-02
interface Ethernet44
   no switchport
   ip address 172.18.0.249/30
 

 

 The rest of the configurations for ONOS-vRouter are the same with one the gateway node setup. 


HA Setup

Basically, ONOS itself provides HA by default when there are multiple instances in the cluster. This section describes how to add a proxy server beyond the ONOS cluster, and make use of it in Neutron as a single access point of the cluster. For the proxy server, we used the HA proxy server (http://www.haproxy.org) here.

...