Test

Failure Scenario

TestON Test name

Roadmap

HA Sanity Test

This tests runs through all the state and functionality checks of the HA Test suite but waits 60 seconds instead of inducing a failure. This is run as a 7 node ONOS cluster.

HAsanity

now

Minority of ONOS Nodes shutdown

Restart 3 of 7 ONOS nodes by gracefully stopping the process once the system is running and stable.

HAkillNodes

now

Minority of ONOS Nodes killed

Restart 3 of 7 ONOS nodes by killing the process once the system is running and stable.

HAstopNodes

now

Entire ONOS cluster restart

Restart 7 of 7 ONOS nodes by killing the process once the system is running and stable.

HAclusterRestart

now

Single node cluster restart

Restart 1 of 1 ONOS nodes by killing the process once the system is running and stable.

HAsingleInstanceRestart

now

Control Network partition

Partition the Control Network by creating IP Table rules once the system is in a stable state.

During Partition:

• ONOS nodes that cannot cluster with a quorum of the cluster will relinquish leadership of any topic and give up any device roles
• Nodes in the majority partition should be able to take up all the work of the cluster and continue to function normally
• Topology is replicated within the sub-cluster the event originated in
• Get flows will only show flows within a sub-cluster
• Intents should only be available in the majority cluster (Intent behavior is not fully defined for the raft implementation)
• Mastership behavior has not been defined for split-brain scenario

After partition is healed:

• Topology is consistent across all nodes and should reflect the current state of the network(including updates during partition)
• Flows view should be consistent across all nodes
• Intents should be available on all nodes including any new intents pushed to the majority sub-clusterpartition
• Mastership should be consistent across all nodes

HAfullNetPartition

now

Partial control network partition

Partially partition the Control Network by creating IP Table rules once the system is in a stable state. (A and B can't talk, but both can talk to C)

• Topology should be consistent across all nodes
• Flows view will show reachable controllers (A sees AC and B see BC and C sees ABC)
• Intents(Wait for Raft implementation, but will depend on which node is the raft leader)
• Mastership(Wait for Raft implementation)

delayed

Change membership of an ONOS cluster at run time

• Start a five node ONOS cluster configured to use a remote metadata file
• Run common state and functionality checks
• Replace two of the five ONOS nodes in the metadata file with two new ONOS nodes
• Check that the two new nodes join the cluster and the two replaced nodes leave
• Run common state and functionality checks

Change the size of an ONOS cluster at run time

• Start a single node ONOS cluster configured to use a remote metadata file
• Run common state and functionality checks
• Scale the cluster up to seven nodes then back down to one node in increments of two nodes.
• After changing cluster size:
  • Check that the two new nodes join the cluster or the two old nodes leave
  • Run common state and functionality checks