Test | Failure Scenario | TestON Test name | Roadmap |
HA Sanity Test | This tests runs through all the state and functionality checks of the HA Test suite but waits 60 seconds instead of inducing a failure. This is run as a 7 node ONOS cluster. | HAsanity | now |
Minority of ONOS Nodes shutdown | Restart 3 of 7 ONOS nodes by gracefully stopping the process once the system is running and stable. | HAkillNodesHAstopNodes | now |
Minority of ONOS Nodes killed | Restart 3 of 7 ONOS nodes by killing the process once the system is running and stable. | HAstopNodesHAkillNodes | now |
Entire ONOS cluster restart | Restart 7 of 7 ONOS nodes by killing the process once the system is running and stable. | HAclusterRestart | now |
Single node cluster restart | Restart 1 of 1 ONOS nodes by killing the process once the system is running and stable. | HAsingleInstanceRestart | now |
Control Network partition | Partition the Control Network by creating IP Table rules once the system is in a stable state. During Partition: - ONOS nodes that cannot cluster with a quorum of the cluster will relinquish leadership of any topic and give up any device roles
- Nodes in the majority partition should be able to take up all the work of the cluster and continue to function normally
After partition is healed: - Topology is consistent across all nodes and should reflect the current state of the network(including updates during partition)
- Flows view should be consistent across all nodes
- Intents should be available on all nodes including any new intents pushed to the majority partition
- Mastership should be consistent across all nodes
| HAfullNetPartition | now |
Dynamic Clustering: Swap nodes | Change membership of an ONOS cluster at run time - Start a five node ONOS cluster configured to use a remote metadata file
- Run common state and functionality checks
- Replace two of the five ONOS nodes in the metadata file with two new ONOS nodes
- Check that the two new nodes join the cluster and the two replaced nodes leave
- Run common state and functionality checks
| HAswapNodes | now |
Dynamic Clustering: Scale up/down | Change the size of an ONOS cluster at run time - Start a single node ONOS cluster configured to use a remote metadata file
- Run common state and functionality checks
- Scale the cluster up to seven nodes then back down to one node in increments of two nodes.
- After changing cluster size:
- Check that the two new nodes join the cluster or the two old nodes leave
- Run common state and functionality checks
| HAscaling | now |