ID | Description | Passing Criteria | TestON Test name | Roadmap |
HA-1 | HA Sanity Test | - Topology, Mastership, Intent and flow data is consistent across a 7 node cluster.
- Basic functionality works (Topology discovery, adding intents,...)
| HATestSanity | now |
HA-2 | Switch Failure | - Topology is updated and intents are recompiled
| All HA Tests | now |
HA-3 | Link Failure | - Topology is updated and intents are recompiled
| All HA Tests | now |
HA-4 | Minority of ONOS Nodes restart | - Data stores continue to function and no information is lost.
- No loss in functionality
| HATestMinorityRestart | now |
HA-5 | Entire ONOS cluster restart | - Persistant data is recovered
- Non-persistant data is rediscovered (if applicable)
- No loss in functionality
| HATestClusterRestart | now |
HA-6 | Single node cluster restart | - Persistant data is recovered
- Non-persistant data is rediscovered (if applicable)
- No loss in functionality
| SingleInstanceHATestRestart | In Progress |
HA-7 | Communication between a device and the ONOS node with mastership | - A new device master is elected from the pool of backups
- No loss in functionality
| | |
HA-8 | Control Network partition | During Partition: - Topology is replicated within the sub-cluster the event originated in
- Get flows will only show flows within a sub-cluster
- Intents should only be available in the majority cluster (Intent behavior is not fully defined for the raft implementation)
- Mastership behavior has not been defined for split-brain scenario
After partition is healed: - Topology is consistent across all nodes and should reflect the current state of the network(including updates during partition)
- Flows view should be consistent across all nodes
- Intents should be available on all nodes including any new intents pushed to the majority sub-cluster
- Mastership should be consistent across all nodes
| | |
HA-9 | Partial network partition (A and B can't talk, but both can talk to C) | - Topology should be consistent across all nodes
- Flows view will show reachable controllers (A sees AC and B see BC and C sees ABC)
- Intents(Wait for Raft implementation, but will depend on which node is the raft leader)
- Mastership(Wait for Raft implementation)
| | |
| Leadership Election | Applications can run for leadership of topics. This service should be safe, stable and fault tolerant. We will check that the service is still functional after failures and there is always only one leader per topic in a cluster. | All HA Tests | Next sprint |
| | | | |
| | | | |