...
1. Refer to the guide(SONA Network Configuration Guide) and write a network configuration for SONA. Place the network-cfg.json under toolsunder tools/package/config/, build package, and then install ONOS. Here's an example cell configuration and commands.
...
Code Block |
---|
|
$ git clone https://github.com/hyunsun/sona-setup.git
$ cd sona-setup |
4. The The first container is ONOS-vRouter. Refer to the guide(SONA Network Configuration Guide) and write a network configuration for vRouter. Name it to it vrouter.json, place under sona-setup and run vrouter.sh sh script, which brings up ONOS container with vRouter application activated. There's a sample named "vrouter.json" under "sona-setup", and note that you don't need to put correct port numbers in the "Ports" section now care about all the port numbers at this time (just leave it as it is in the sample file). We're going to fix the port numbers later in Step 7. Just check out the IP, MAC addresses, and also the eth0 interface IP address of the container. These are required to configure Quagga container.
Code Block |
---|
# modifyedit vrouter.json
sona-setup$ vrouter.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e5ac67e62bbb onosproject/onos:1.6 "./bin/onos-service" 9 days ago Up 9 days 6653/tcp, 8101/tcp, 8181/tcp, 9876/tcp onos-vrouter |
5. Next, edit Modify volumes/gateway/zebra.conf
and volumes/gateway/bgpd.conf
as you want. Here are samples of the config files. Note that fpm connection ip
in zebra.conf
should be the eth0 interface IP address of onos-vrouter container, assigned by Docker. Run Quagga container with those config files. The IP address comes with the command would be equals to router-id
in bgpd.conf
.
Code Block |
---|
title | volumes/gateway/bgpd.conf |
---|
|
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname gateway-01
password zebra
!
router bgp 65101
bgp router-id 172.18.0.254
timers bgp 3 9
neighbor 172.18.0.1253 remote-as 65100
neighbor 172.18.0.1253 ebgp-multihop
neighbor 172.18.0.1253 timers connect 5
neighbor 172.18.0.1253 advertisement-interval 5
network 172.27.0.0/24
!
log file /var/log/quagga/bgpd.log |
Code Block |
---|
title | volumes/gateway/zebra.conf |
---|
|
!
hostname gateway-01
password zebra
!
fpm connection ip 172.17.0.2 port 2620 |
Note that the FPM connection IP should be the ONOS-vRouter container's eth0 IP address. Docker increments the IP address whenever a new container created, so normally it would be 172.17.0.2 but if you have problem with FPM connection later, please check this IP is correct.
Code Block |
---|
|
$ ./quagga.sh -- |
Code Block |
---|
|
$ ./quagga.sh --name=gateway-01 --ip=172.18.0.254/30 --mac=fe:00:00:00:00:01 |
...
Code Block |
---|
title | volumes/router/bgpd.conf |
---|
|
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname router-01
password zebra
!
router bgp 65100
bgp router-id 172.18.0.1253
timers bgp 3 9
neighbor 172.18.0.254 remote-as 65101
neighbor 172.18.0.254 ebgp-multihop
neighbor 172.18.0.254 timers connect 5
neighbor 172.18.0.254 advertisement-interval 5
neighbor 172.18.0.254 default-originate
!
log file /var/log/quagga/bgpd.log |
...
Code Block |
---|
|
$ ./quagga.sh --name=router-01 --ip=172.18.0.1253/30 --mac=fa:00:00:00:00:01 --external-router |
If you check the result of ovs-vsctl show
, there should be a new port named quagga-router
on br-router
bridge.
7. Once all the containers are up and running, check ports
result from It's time to fix the port numbers in the ONOS-vRouter . If any port number does not match to the ones in vrouter.json
, modify the config file with the correct port numbers, and just re-run the vrouter.sh. It actually happens often since you may re-create Quagga containers multiple times to fix the Quagga config files or for some other reasons, and OVS increases port number whenever a new port is added to a bridge. network configuration (vrouter.json file). Now you should be able to find all OpenFlow port number required to set the configuration with the "ovs-ofctl show br-router" command. Please fix the port numbers appropriately as below.
Code Block |
---|
$ sshsudo ovs-pofctl 8101 karaf@172.17.0.2
# password is karaf
onos> ports
id=of:00000000000000b1, available=true, role=MASTER, type=SWITCH, mfr=Nicira, Inc., hw=Open vSwitch, sw=2.5.0, serial=None, driver=softrouter, channelId=172.17.0.1:58292, managementAddress=172.17.0.1, name=of:00000000000000b1, protocol=OF_13
port=local, state=disabled, type=copper, speed=0 , portName=br-router, portMac=e6:a0:79:f9:d1:4a
port=1, state=enabled, type=copper, speed=0 , portName=patch-rout, portMac=fe:da:85:15:b1:bf
port=24, state=enabled, type=copper, speed=10000 , portName=quagga, portMac=06:96:1b:36:32:77
port=25, state=enabled, type=copper, speed=10000 , portName=quagga-router, portMac=ea:1e:71:d1:fd:81show br-router
OFPT_FEATURES_REPLY (xid=0x2): dpid:00000000000000b1
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(patch-rout): addr:96:67:91:a4:24:f7
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
24(quagga): addr:52:b2:b0:0f:b1:5b
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
25(quagga-router): addr:1a:64:b1:37:c7:a6
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(br-router): addr:5a:59:0a:9f:0f:4c
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 |
Code Block |
---|
|
"apps" : { |
Code Block |
---|
|
"apps" : {
"org.onosproject.router" : {
"router" : {
"controlPlaneConnectPoint" : "of:00000000000000b1/24",
"ospfEnabled" : "true",
"interfaces" : [ "b1-1", "b1-2" ]
}
}
},
"ports" : {
"of:00000000000000b1/25" : {
"interfaces" : [
"org.onosproject.router" : {
"name"router" : "b1-1",{
"ipscontrolPlaneConnectPoint" : [ "172.18.0.254"of:00000000000000b1/24" ],
"macospfEnabled" : "fe:00:00:00:00:01"true",
} "interfaces" : [ "b1-1", "b1-2" ]
]}
},
},
"ports" : {
"of:00000000000000b1/125" : {
"interfaces" : [
{
"name" : "b1-21",
"ips" : [ "172.2718.0.254/2430" ],
"mac" : "fe:00:00:00:00:01"
}
]
}
},
"hosts" : {
"fe:00:00:00:00:02/-of:00000000000000b1/1" : {
"basicinterfaces" : {[
"ips": ["172.27.0.1"],
{
"locationname" : "of:00000000000000b1/1"b1-2",
}
}
"ips" : } |
- Line 4: Device ID and port number of the port with portName=
quagga
-> controlPlaneConnectPoint
- Line 11: Device ID and port number of the port with portName=
quagga-router
or other actual uplink port.
...
...
...
" ],
"mac" : "fe:00:00:00:00:01"
}
]
}
},
"hosts" : {
"fe:00:00:00:00:02/-1" : {
"basic": {
"ips": ["172.27.0.1"],
"location": "of:00000000000000b1/1"
}
}
} |
- Line 4: Device ID and port number of the port with portName=
quagga
-> controlPlaneConnectPoint
- Line 11: Device ID and port number of the port with portName=
quagga-router
or other actual uplink port.
If you have a floating range, 172.27.0.0/24 in this example, check the following configurations also.
- Line 20: (optional interface config for floating IP address range) Device ID and port number of the port with portName=
patch-rout
- Line 34: (optional interface config for floating IP gateway) Device ID and port number of the port with portName=
patch-rout
And then restart the vrouter container again by simply running vrouter.sh.
Code Block |
---|
sona-setup$ vrouter.sh
sona-setup$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3cdbf6a76d10 onosproject/onos:1.6 "./bin/onos-service" About a minute ago Up 59 seconds 6653/tcp, 8101/tcp, 8181/tcp, 9876/tcp onos-vrouter
d5763d29afc4 hyunsun/quagga-fpm "/usr/bin/supervisord" 3 minutes ago Up 3 minutes 179/tcp, 2601/tcp, 2605/tcp router-01
74f429f98174 hyunsun/quagga-fpm "/usr/bin/supervisord" 4 minutes ago Up 4 minutes 179/tcp, 2601/tcp, 2605/tcp gateway-01 |
8. If everything's right, check fpm-connections
, hosts
and routes
. 172.18.0.253
is the external default gateway in this example. If you added interface and host for floating IP range, you should be able to see the host in the list.
Code Block |
---|
onos> hosts
id=FA:00:00:00:00:01/None, mac=FA:00:00:00:00:01, location=of:00000000000000b1/25, vlan=None, ip(s)=[172.18.0.1]
id=FE:00:00:00:00:01/None, mac=FE:00:00:00:00:01, location=of:00000000000000b1/24, vlan=None, ip(s)=[172.18.0.254]
id=FE:00:00:00:00:02/None, mac=FE:00:00:00:00:02, location=of:00000000000000b1/1, vlan=None, ip(s)=[172.27.0.253], name=FE:00:00:00:00:02/None
onos> fpm-connections
172.17.0.3:52332 connected since 6m ago
onos> next-hops
ip=172.18.0.253, mac=FA:00:00:00:00:01, numRoutes=1
onos> routes
Table: ipv4
Network Next Hop
0.0.0.0/0 172.18.0.253
Total: 1
Table: ipv6
Network Next Hop
Total: 0 |
8. Add route for floating IP range manually and check the route is added.
Code Block |
---|
onos> route-add 172.27.0.0/24 172.27.0.1
onos> routes
Table: ipv4
Network Next Hop
0.0.0.0/0 172.18.0.1
172.27.0.0/24 172.27.0.1
Total: 2
Table: ipv6
Network Next Hop
Total: 0
onos> next-hops
ip=172.18.0.253, mac=FA:00:00:00:00:01, numRoutes=1
ip=172.27.0.1, mac=FE:00:00:00:00:02, numRoutes=1 |
9. Now you should be able to see the gateway node is in COMPLETE state when you re-trigger node initialization. You can either run command openstack-node-init gateway-01 or push the network configuration file again.
Code Block |
---|
onos> openstack-node-init gateway-01 |
Or
Code Block |
---|
title | how to push network config |
---|
|
$ curl --user onos:rocks -X POST -H "Content-Type: application/json" http://10.134.231.29:8181/onos/v1/network/configuration/ -d @network-cfg.json |
Code Block |
---|
onos> openstack-nodes
hostname=compute-01, type=COMPUTE, managementIp=10.134.231.30, dataIp=10.134.34.222, intBridge=of:00000000000000a1, routerBridge=Optional.empty init=COMPLETE
hostname=compute-02, type=COMPUTE, managementIp=10.134.231.31, dataIp=10.134.34.223, intBridge=of:00000000000000a2, routerBridge=Optional.empty init=COMPLETE
hostname=gateway-01, type=GATEWAY, managementIp=10.134.231.32, dataIp=10.134.33.224, intBridge=of:00000000000000a3, routerBridge=Optional[of:00000000000000b1] init=COMPLETE
hostname=gateway-02, type=GATEWAY, managementIp=10.134.231.33, dataIp=10.134.33.225, intBridge=of:00000000000000a4, routerBridge=Optional[of:00000000000000b2] init=COMPLETE
Total 4 nodes |
Multiple Gateway Nodes Setup
SONA allows multiple gateway nodes for HA as well as scalability. Here's another example of multiple gateway nodes and external upstream router. In this scenario, each gateway node should have unique IP and MAC address for peering so that the upstream router can handle each of them as a different router. ONOS scalable gateway application is responsible for taking upstream packet through one of the gateway nodes, and the upstream router is responsible for taking downstream packet through one of the gateway nodes.
The following is an example Quagga configuration of the second gateway node.
Code Block |
---|
title | volumes/gateway/bgpd.conf |
---|
|
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname gateway-02
password zebra
!
router bgp 65101
bgp router-id 172.18.0.250
timers bgp 3 9
neighbor 172.18.0.249 remote-as 65100
neighbor 172.18.0.249 ebgp-multihop
neighbor 172.18.0.249 timers connect 5
neighbor 172.18.0.249 advertisement-interval 5
network 172.27.0.0/24
!
log file /var/log/quagga/bgpd.log |
Code Block |
---|
title | volumes/gateway/zebra.conf |
---|
|
!
hostname gateway-02
password zebra
!
fpm connection ip 172.17.0.2 port 2620 |
Once you are done with the configurations for the second Quaaga, run quagga.sh script to bring up Quagga container with the MAC address different from the one used for the first gateway node Quagga container.
Code Block |
---|
$ ./quagga.sh --name=gateway-02 --ip=172.18.0.250/30 --mac=fe:00:00:00:00:03 |
You also need to run ONOS-vRouter in the second gateway node. Don't forget to set correct IP and MAC address, 172.18.0.250 and fe:00:00:00:00:03, instead of 172.18.0.254 and fe:00:00:00:00:01 in the above network config example for successful peering.
Code Block |
---|
"apps" : {
"org.onosproject.router" : {
"router" : {
"controlPlaneConnectPoint" : "of:00000000000000b1/2",
"ospfEnabled" : "true",
"interfaces" : [ "b1-1", "b1-2" ]
}
}
},
"ports" : {
"of:00000000000000b1/3" : {
"interfaces" : [
{
"name" : "b1-1",
"ips" : [ "172.18.0.250/30" ],
"mac" : "fe:00:00:00:00:03"
|
- Line 20: (optional interface config for floating IP address range) Device ID and port number of the port with portName=
patch-rout
- Line 34: (optional interface config for floating IP gateway) Device ID and port number of the port with portName=
patch-rout
Once you fix the vrouter.json file, re-create onos-vrouter container with the updated configuration. vrouter.sh script will take care of removing the existing container.
Code Block |
---|
sona-setup$ vrouter.sh |
8. If everything's right, check fpm-connections
, hosts
and routes
. 172.18.0.1
is the external default gateway in this example. If you added interface and host for floating IP range, you should be able to see the host in the list.
Code Block |
---|
onos> hosts
id=FA:00:00:00:00:01/None, mac=FA:00:00:00:00:01, location=of:00000000000000b1/25, vlan=None, ip(s)=[172.18.0.1]
id=FE:00:00:00:00:01/None, mac=FE:00:00:00:00:01, location=of:00000000000000b1/24, vlan=None, ip(s)=[172.18.0.254]
id=FE:00:00:00:00:02/None, mac=FE:00:00:00:00:02, location=of:00000000000000b1/1, vlan=None, ip(s)=[172.27.0.1], name=FE:00:00:00:00:02/None
onos> fpm-connections
172.17.0.2:52332 connected since 6m ago
onos> next-hops
ip=172.18.0.1, mac=FA:00:00:00:00:01, numRoutes=1
onos> routes
Table: ipv4
Network Next Hop}
0.0.0.0/0 172.18.0.1]
Total: 1
Table: ipv6
Network},
Next Hop
Total: 0 |
9. Add route for floating IP range manually and check the route is added.
Code Block |
---|
onos> route-add 172.27.0.0/24 172.27.0.1
onos> routes
Table: ipv4"of:00000000000000b1/1" : {
Network "interfaces" : [
Next Hop
0.0.0.0/0 172.18.0.1{
172.27.0.0/24 172.27.0.1
"name" Total: "b1-2
Table: ipv6
",
Network Next Hop
Total: 0
onos> next-hops
ip=172.18.0.1, mac=FA:00:00:00:00:01, numRoutes=1
ip= "ips" : [ "172.27.0.1, mac=FE:00:00:00:00:02, numRoutes=1 |
10. Now you should be able to see the gateway node is in COMPLETE state when you re-trigger node initialization. You can either run command openstack-node-init gateway-01 or push the network configuration file again.
Code Block |
---|
onos> openstack-node-init gateway-01
onos> openstack-node-init gateway-02 |
Or
Code Block |
---|
title | how to push network config |
---|
|
$ curl --user onos:rocks -X POST -H "Content-Type: application/json" http://10.134.231.29:8181/onos/v1/network/configuration/ -d @network-cfg.json |
Code Block |
---|
onos> openstack-nodes
hostname=compute-01, type=COMPUTE, managementIp=10.134.231.30, dataIp=10.134.34.222, intBridge=of:00000000000000a1, routerBridge=Optional.empty init=COMPLETE
hostname=compute-02, type=COMPUTE, managementIp=10.134.231.31, dataIp=10.134.34.223, intBridge=of:00000000000000a2, routerBridge=Optional.empty init=COMPLETE
hostname=gateway-01, type=GATEWAY, managementIp=10.134.231.32, dataIp=10.134.33.224, intBridge=of:00000000000000a3, routerBridge=Optional[of:00000000000000b1] init=COMPLETE
hostname=gateway-02, type=GATEWAY, managementIp=10.134.231.33, dataIp=10.134.33.225, intBridge=of:00000000000000a4, routerBridge=Optional[of:00000000000000b2] init=COMPLETE
Total 4 nodes |
Multiple Gateway Nodes Setup
SONA allows multiple gateway nodes for HA as well as scalability. Here's another example of multiple gateway nodes and external upstream router. In this scenario, each gateway node should have unique IP and MAC address for peering so that the upstream router can handle each of them as a different router. ONOS scalable gateway application is responsible for taking upstream packet through one of the gateway nodes, and the upstream router is responsible for taking downstream packet through one of the gateway nodes.
The following is an example Quagga configuration of the second gateway node.
Code Block |
---|
title | volumes/gateway/bgpd.conf |
---|
|
! -*- bgp -*-
!
! BGPd sample configuration file
!
!
hostname gateway-02
password zebra
!
router bgp 65101
bgp router-id 172.18.0.250
timers bgp 3 9
neighbor 172.18.0.1 remote-as 65100
neighbor 172.18.0.1 ebgp-multihop
neighbor 172.18.0.1 timers connect 5
neighbor 172.18.0.1 advertisement-interval 5
network 172.27.0.0/24
!
log file /var/log/quagga/bgpd.log |
Code Block |
---|
title | volumes/gateway/zebra.conf |
---|
|
!
hostname gateway-02
password zebra
!
fpm connection ip 172.17.0.2 port 2620 |
Once you are done with the configurations for the second Quaaga, run quagga.sh script to bring up Quagga container with the MAC address different from the one used for the first gateway node Quagga container.
...
254/24" ],
"mac" : "fe:00:00:00:00:03"
}
]
}
},
"hosts" : {
"fe:00:00:00:00:02/-1" : {
"basic": {
"ips": ["172.27.0.1"],
"location": "of:00000000000000b1/1"
}
}
} |
...
Now configure the upstream router as below.
...
Code Block |
---|
#routed port connected to gateway-01
interface Ethernet43
no switchport
ip address 172.18.0.253/30
#routed port connected to gateway-02
interface Ethernet44
no switchport
ip address 172.18.0.249/30
|
...
HA Setup
Basically, ONOS itself provides HA by default when there are multiple instances in the cluster. This section describes how to add a proxy server beyond the ONOS cluster, and make use of it in Neutron as a single access point of the cluster. For the proxy server, we used the HA proxy server (http://www.haproxy.org) here.
...