Page History

...

...

<img src="https://jenkins.onosproject.org/view/QA/job/postjob-VM/lastSuccessfulBuild/artifact/HAscaling_master_20-builds_graph.jpg", alt="HAscaling", style="width:525px;height:350px;border:0">

commit b53d626fb03195129c0ec0802e1f953f85dcbef8 (HEAD, origin/master, origin/HEAD, master)
Author: pierventre [pier@opennetworking.org]
AuthorDate: Wed Sep 22 11:24:38 2021 +0200
Commit: Pier Luigi Ventre [pier@opennetworking.org]
CommitDate: Thu Sep 23 07:38:36 2021 +0000

[SDFAB-612] Cluster not ready when using recent tost master images
--
(cherry picked from commit ec0e942320ef27932b4173172055151015261d7c)

Case 1: Constructing test variables and building ONOS package - PASS

For loading from params file, and pull and build the latest ONOS package

• 1.1 Constructing test variables - PASS
• 1.2 Apply cell to environment - PASS
• 1.3 Uninstalling Atomix - PASS
• 1.4 Uninstalling ONOS package - PASS
• 1.5 Starting Mininet - PASS
• 1.6 Creating ONOS package - PASS
• 1.7 Installing Atomix - PASS
• 1.8 Installing ONOS package - PASS
• 1.9 Set up ONOS secure SSH - PASS
• 1.10 Checking ONOS service - PASS
• 1.11 Starting ONOS CLI sessions - PASS
• 1.12 Checking ONOS nodes - PASS
• 1.13 Checking ONOS applications - PASS
• 1.14 Checking ONOS nodes - PASS
• 1.15 Activate apps defined in the params file - No Result
• 1.16 Set ONOS configurations - PASS
• 1.17 Check app ids - PASS
• 1.18 Set logging levels - PASS

Case 2: Assigning devices to controllers - PASS

Assign switches to ONOS using 'ovs-vsctl' and check that an ONOS node becomes the master of the device.

• 2.1 Assign switches to controllers - PASS

Case 8: Compare ONOS Topology view to Mininet topology - PASS

Compare topology objects between Mininet and ONOS

• 8.1 Comparing ONOS topology to MN topology - PASS
• 8.2 Hosts view is consistent across all ONOS nodes - PASS
• 8.3 Hosts information is correct - PASS
• 8.4 Host attachment points to the network - PASS
• 8.5 Clusters view is consistent across all ONOS nodes - PASS
• 8.6 There is only one SCC - PASS
• 8.7 Device information is correct - PASS
• 8.8 Links are correct - PASS
• 8.9 Hosts are correct - PASS
• 8.10 Checking ONOS nodes - PASS

Case 21: Assigning Controller roles for switches - PASS

Check that ONOS is connected to each device. Then manually assign mastership to specific ONOS nodes using 'device-role'

• 21.1 Assign mastership of switches to specific controllers - PASS
• 21.2 Check mastership was correctly assigned - PASS

Case 3: Adding host Intents - PASS

Discover hosts by using pingall then assign predetermined host-to-host intents. After installation, check that the intent is distributed to all nodes and the state is INSTALLED

• 3.1 Install reactive forwarding app - PASS
• 3.2 Check app ids - PASS
• 3.3 Discovering Hosts( Via pingall for now ) - PASS
• 3.4 Uninstall reactive forwarding app - PASS
• 3.5 Check app ids - PASS
• 3.6 Add host intents via cli - PASS
• 3.7 Intent Anti-Entropy dispersion - PASS

Case 8: Compare ONOS Topology view to Mininet topology - PASS

Compare topology objects between Mininet and ONOS

• 8.1 Comparing ONOS topology to MN topology - PASS
• 8.2 Hosts view is consistent across all ONOS nodes - PASS
• 8.3 Hosts information is correct - PASS
• 8.4 Host attachment points to the network - PASS
• 8.5 Clusters view is consistent across all ONOS nodes - PASS
• 8.6 There is only one SCC - PASS
• 8.7 Device information is correct - PASS
• 8.8 Links are correct - PASS
• 8.9 Hosts are correct - PASS
• 8.10 Checking ONOS nodes - PASS

Case 4: Verify connectivity by sending traffic across Intents - PASS

Ping across added host intents to check functionality and check the state of the intent

• 4.1 Check Intent state - PASS
• 4.2 Ping across added host intents - PASS
• 4.3 Check leadership of topics - PASS
• 4.4 Wait a minute then ping again - PASS

Case 5: Setting up and gathering data for current state - PASS

• 5.1 Check that each switch has a master - PASS
• 5.2 Get the Mastership of each switch from each controller - No Result
• 5.3 Read device roles from ONOS - PASS
• 5.4 Check for consistency in roles from each controller - PASS
• 5.5 Get the intents from each controller - PASS
• 5.6 Check for consistency in Intents from each controller - PASS
• 5.7 Get the flows from each controller - PASS
• 5.8 Check for consistency in Flows from each controller - PASS
• 5.9 Get the OF Table entries - No Result
• 5.10 Start continuous pings - No Result
• 5.11 Collecting topology information from ONOS - No Result
• 5.12 Host view is consistent across ONOS nodes - PASS
• 5.13 Each host has an IP address - PASS
• 5.14 Cluster view is consistent across ONOS nodes - PASS
• 5.15 Cluster view correct across ONOS nodes - PASS
• 5.16 Comparing ONOS topology to MN - PASS
• 5.17 Device information is correct - PASS
• 5.18 Links are correct - PASS
• 5.19 Hosts are correct - PASS

Case 14: Start Leadership Election app - PASS

• 14.1 Install leadership election app - PASS
• 14.2 Run for election on each node - PASS
• 14.3 Active node was elected leader? - PASS

Case 16: Install Primitives app - PASS

• 16.1 Install Primitives app - PASS

Case 17: Check for basic functionality with distributed primitives - PASS

Test the methods of the distributed primitives (counters and sets) throught the cli

• 17.1 Increment then get a default counter on each node - PASS
• 17.2 Get then Increment a default counter on each node - PASS
• 17.3 Counters we added have the correct values - PASS
• 17.4 Add -8 to then get a default counter on each node - PASS
• 17.5 Add 5 to then get a default counter on each node - PASS
• 17.6 Get then add 5 to a default counter on each node - PASS
• 17.7 Counters we added have the correct values - PASS
• 17.8 Distributed Set get - PASS
• 17.9 Distributed Set size - PASS
• 17.10 Distributed Set add() - PASS
• 17.11 Distributed Set addAll() - PASS
• 17.12 Distributed Set contains() - PASS
• 17.13 Distributed Set containsAll() - PASS
• 17.14 Distributed Set remove() - PASS
• 17.15 Distributed Set removeAll() - PASS
• 17.16 Distributed Set addAll() - PASS
• 17.17 Distributed Set clear() - PASS
• 17.18 Distributed Set addAll() - PASS
• 17.19 Distributed Set retain() - PASS
• 17.20 Partitioned Transactional maps put - PASS
• 17.21 Partitioned Transactional maps get - PASS
• 17.22 Get the value of a new value - PASS
• 17.23 Atomic Value set() - PASS
• 17.24 Get the value after set() - PASS
• 17.25 Atomic Value compareAndSet() - PASS
• 17.26 Get the value after compareAndSet() - PASS
• 17.27 Atomic Value getAndSet() - PASS
• 17.28 Get the value after getAndSet() - PASS
• 17.29 Atomic Value destory() - PASS
• 17.30 Get the value after destroy() - PASS
• 17.31 Work Queue add() - PASS
• 17.32 Check the work queue stats - PASS
• 17.33 Work Queue addMultiple() - PASS
• 17.34 Check the work queue stats - PASS
• 17.35 Work Queue takeAndComplete() 1 - PASS
• 17.36 Check the work queue stats - PASS
• 17.37 Work Queue takeAndComplete() 2 - PASS
• 17.38 Check the work queue stats - PASS
• 17.39 Work Queue destroy() - PASS
• 17.40 Check the work queue stats - PASS

Case 6: Scale the number of nodes in the ONOS cluster - PASS

• 6.1 Checking ONOS Logs for errors - No Result
• 6.2 Scaling from 1 to 3 nodes - No Result
• 6.3 Start new nodes - PASS
• 6.4 Set up ONOS secure SSH - PASS
• 6.5 Checking ONOS service - PASS
• 6.6 Checking ONOS nodes

<iframe src="https://onos-jenkins.onlab.us/job/HAscaling/plot/Plot-HA/getPlot?index=1&width=500&height=300"noborder="0" width="500" height="300" scrolling="yes" seamless="seamless"></iframe>

Case 1: Setting up test environment - PASS

Setup the test environment including installing ONOS, starting Mininet and ONOScli sessions.

• 1.1 Create cell file - No Result
• 1.2 Applying cell variable to environment - PASS
• 1.3 Verify connectivity to cell - PASS
• 1.4 Setup server for cluster metadata file - PASS
• 1.5 Generate initial metadata file - PASS
• 1.6 Starting Mininet - PASS
• 1.7 Git checkout and pull master - No Result
• 1.8 Using mvn clean install - PASS
• 1.9 Copying backup config files - PASS
• 1.10 Creating ONOS package - PASS
• 1.11 Installing ONOS package - PASS
• 1.12 Set up ONOS secure SSH - PASS
• 1.13 Checking if ONOS is up yet - PASS
• 1.14 Starting ONOS CLI sessions - PASS
• 1.15 Checking ONOS nodes - PASS
• 1.16 Activate apps defined in the params file - No Result
• 1.17 Set ONOS configurations - PASS
• 1.18 App Ids check - PASS

Case 2: Assigning devices to controllers - PASS

Assign switches to ONOS using 'ovs-vsctl' and check that an ONOS node becomes the master of the device.

• 2.1 Assign switches to controllers - PASS

Case 8: Compare ONOS Topology view to Mininet topology - PASS

Compare topology objects between Mininet and ONOS

• 8.1 Comparing ONOS topology to MN topology - PASS
• 8.2 Hosts view is consistent across all ONOS nodes - PASS
• 8.3 Hosts information is correct - PASS
• 8.4 Host attachment points to the network - PASS
• 8.5 Clusters view is consistent across all ONOS nodes - PASS
• 8.6 There is only one SCC - PASS
• 8.7 Device information is correct - PASS
• 8.8 Links are correct - PASS 8.8 Links are correct
• 8.9 Hosts are correct - PASS
• 8.10 Checking ONOS nodes - PASS

Case 7: Running ONOS Constant State Tests - PASS

• 7.1 Check that each switch has a master - PASS
• 8.9 Hosts are correct 7.2 Read device roles from ONOS - PASS 8.10 Checking ONOS nodes
• 7.3 Check for consistency in roles from each controller - PASS

Case 21: Assigning Controller roles for switches - PASS

Check that ONOS is connected to each device. Then manually assign mastership to specific ONOS nodes using 'device-role'

• 7.4 Get the intents from each controller - PASS
• 7.5 Check for consistency in Intents from each controller - PASS
• 7.6 Compare current intents with intents before the scaling - PASS
• 7.7 Get the OF Table entries and compare to before component scaling - PASS
• 7.8 Leadership Election is still functional
• 21.1 Assign mastership of switches to specific controllers - PASS
• 21.2 Check mastership was correctly assigned - PASS

Case

...

4: Verify connectivity by sending traffic across Intents - PASS

Discover hosts by using pingall then assign predetermined host-to-host intents. After installation, check that the intent is distributed to all nodes and the state is INSTALLED

Ping across added host intents to check functionality and check the state of the intent

• 4.1 Check Intent state 3.1 Install reactive forwarding app - PASS 3
• 4.2 Ping across added host intents - PASS
• 4.3 Check leadership of topics Check app ids - PASS 3.3 Discovering Hosts( Via pingall for now
• ) 4.4 Wait a minute then ping again - PASS
• 3.4 Uninstall reactive forwarding app - PASS

Case 15: Check that Leadership Election is still functional - PASS

• 15.1 Run for election on each node 3.5 Check app ids - PASS 3.6 Add host intents via cli
• 15.2 Check that each node shows the same leader and candidates - PASS
• 3.7 Intent Anti-Entropy dispersion 15.3 Find current leader and withdraw - PASS

Case 8: Compare ONOS Topology view to Mininet topology - PASS

Compare topology objects between Mininet and ONOS

• 15.4 Check that a new node was elected leader - PASS
• 15.5 Check that that new leader was the candidate of old leader 8.1 Comparing ONOS topology to MN topology - PASS 8.2 Hosts view is consistent across all ONOS
• nodes 15.6 Run for election on old leader( just so everyone is in the hat ) - PASS
• 8.3 Hosts information is correct - PASS
• 8.4 Host attachment points to the network - PASS
• 8.5 Clusters view is consistent across all ONOS nodes - PASS
• 8.6 There is only one SCC - PASS
• 8.7 Device information is correct - PASS
• 8.8 Links are correct - PASS
• 8.9 Hosts are correct - PASS
• 8.10 Checking ONOS nodes - PASS

Case 4: Verify connectivity by sending traffic across Intents - PASS

Ping across added host intents to check functionality and check the state of the intent

• 4.1 Check Intent state - PASS
• 4.2 Ping across added host intents - PASS
• 4.3 Check leadership of topics - PASS
• 4.4 Wait a minute then ping again - PASS

Case 5: Setting up and gathering data for current state - PASS

• 5.1 Check that each switch has a master - PASS
• 5.2 Get the Mastership of each switch from each controller - PASS
• 5.3 Check for consistency in roles from each controller - PASS
• 5.4 Get the intents from each controller - PASS
• 5.5 Check for consistency in Intents from each controller - PASS
• 5.6 Get the flows from each controller - PASS
• 5.7 Check for consistency in Flows from each controller - PASS
• 5.8 Get the OF Table entries - No Result
• 5.9 Start continuous pings - No Result
• 5.10 Collecting topology information from ONOS - No Result
• 5.11 Host view is consistent across ONOS nodes - PASS
• 5.12 Each host has an IP address - PASS
• 5.13 Cluster view is consistent across ONOS nodes - PASS
• 5.14 Cluster view correct across ONOS nodes - PASS
• 5.15 Comparing ONOS topology to MN - PASS
• 5.16 Device information is correct - PASS
• 5.17 Links are correct - PASS
• 5.18 Hosts are correct - PASS

Case 14: Start Leadership Election app - PASS

• 14.1 Install leadership election app - PASS
• 14.2 Run for election on each node - PASS
• 14.3 First node was elected leader - PASS

Case 16: Install Primitives app - PASS

• 16.1 Install Primitives app - PASS

Case 17: Check for basic functionality with distributed primitives - PASS

Test the methods of the distributed primitives (counters and sets) throught the cli

• 17.1 Increment then get a default counter on each node - PASS
• 17.2 Get then Increment a default counter on each node - PASS
• 17.3 Counters we added have the correct values - PASS
• 17.4 Add -8 to then get a default counter on each node - PASS
• 17.5 Add 5 to then get a default counter on each node - PASS
• 17.6 Get then add 5 to a default counter on each node - PASS
• 17.7 Counters we added have the correct values - PASS
• 17.8 Distributed Set get - PASS
• 17.9 Distributed Set size - PASS
• 17.10 Distributed Set add() - PASS
• 17.11 Distributed Set addAll() - PASS
• 17.12 Distributed Set contains() - PASS
• 17.13 Distributed Set containsAll() - PASS
• 17.14 Distributed Set remove() - PASS
• 17.15 Distributed Set removeAll() - PASS
• 17.16 Distributed Set addAll() - PASS
• 17.17 Distributed Set clear() - PASS
• 17.18 Distributed Set addAll() - PASS
• 17.19 Distributed Set retain() - PASS
• 17.20 Partitioned Transactional maps put - PASS
• 17.21 Partitioned Transactional maps get - PASS

Case 6: Scale the number of nodes in the ONOS cluster - FAIL

• 6.1 Checking ONOS Logs for errors - PASS
• 6.2 Start new nodes - PASS
• 6.3 Checking if ONOS is up yet - PASS
• 6.4 Starting ONOS CLI sessions - PASS
• 6.5 Checking ONOS nodes - FAIL
• Failed to rerun for election

Case 8: Compare ONOS Topology view to Mininet topology - PASS

Compare topology objects between Mininet and ONOS

• 8.1 Comparing ONOS topology to MN topology - PASS
• 8.2 Hosts view is consistent across all ONOS nodes - PASS
• 8.3 Hosts information is correct - PASS
• 8.4 Host attachment points to the network - PASS
• 8.5 Clusters view is consistent across all ONOS nodes - PASS
• 8.6 There is only one SCC - PASS
• 8.7 Device information is correct - PASS
• 8.8 Links are correct - PASS
• 8.9 Hosts are correct - PASS
• 8.10 Checking ONOS nodes - PASS

Case 7: Running ONOS Constant State Tests - FAIL

• 7.1 Check that each switch has a master - PASS
• 7.2 Read device roles from ONOS - PASS
• 7.3 Check for consistency in roles from each controller - PASS
• 7.4 Get the intents and compare across all nodes - PASS
• 7.5 Check for consistency in Intents from each controller - PASS
• 7.6 Compare current intents with intents before the scaling - PASS
• 7.7 Get the OF Table entries and compare to before component scaling - PASS
• 7.8 Leadership Election is still functional - FAIL
• Something went wrong with Leadership election

Case 4: Verify connectivity by sending traffic across Intents - PASS

Ping across added host intents to check functionality and check the state of the intent

• 4.1 Check Intent state - PASS
• 4.2 Ping across added host intents - PASS
• 4.3 Check leadership of topics - PASS
• 4.4 Wait a minute then ping again - PASS

Case 15: Check that Leadership Election is still functional - FAIL

• 15.1 Run for election on each node - PASS
• 15.2 Check that each node shows the same leader and candidates - PASS
• 15.3 Find current leader and withdraw - PASS
• 15.4 Check that a new node was elected leader - FAIL
• Something went wrong with Leadership election
• 15.5 Check that that new leader was the candidate of old leader - FAIL
• Incorrect Candidate Elected
• 15.6 Run for election on old leader( just so everyone is in the hat ) - PASS
• 15.7 Check that oldLeader is a candidate, and leader if only 1 node - PASS

Case 17: Check for basic functionality with distributed primitives - FAIL

Test the methods of the distributed primitives (counters and sets) throught the cli

• 17.1 Increment then get a default counter on each node - FAIL
• Error incrementing default counter
• 17.2 Get then Increment a default counter on each node - FAIL
• Error incrementing default counter
• 17.3 Counters we added have the correct values - FAIL
• Added counters are incorrect
• 17.4 Add -8 to then get a default counter on each node - FAIL
• Error incrementing default counter
• 17.5 Add 5 to then get a default counter on each node - FAIL
• Error incrementing default counter
• 17.6 Get then add 5 to a default counter on each node - FAIL
• Error incrementing default counter
• 17.7 Counters we added have the correct values - FAIL
• Added counters are incorrect
• 17.8 Distributed Set get - FAIL
• Set elements are incorrect
• 17.9 Distributed Set size - FAIL
• Set sizes are incorrect
• 17.10 Distributed Set add() - FAIL
• Set add was incorrect
• 17.11 Distributed Set addAll() - FAIL
• Set addAll was incorrect
• 17.12 Distributed Set contains() - FAIL
• Set contains failed
• 17.13 Distributed Set containsAll() - PASS
• 17.14 Distributed Set remove() - FAIL
• Set remove was incorrect
• 17.15 Distributed Set removeAll() - FAIL
• Set removeAll was incorrect
• 17.16 Distributed Set addAll() - FAIL
• Set addAll was incorrect
• 17.17 Distributed Set clear() - FAIL
• Set clear was incorrect
• 17.18 Distributed Set addAll() - FAIL
• Set addAll was incorrect
• 17.19 Distributed Set retain() - FAIL
• Set retain was incorrect
• 17.20 Partitioned Transactional maps put - PASS
• 17.21 Partitioned Transactional maps get - FAIL
• Partitioned Transactional Map values incorrect

Case 6: Scale the number of nodes in the ONOS cluster - FAIL

...

• 15.7 Check that oldLeader is a candidate, and leader if only 1 node - No Result

Case 17: Check for basic functionality with distributed primitives - PASS

Test the methods of the distributed primitives (counters and sets) throught the cli

• 17.1 Increment then get a default counter on each node - PASS
• 17.2 Get then Increment a default counter on each node - PASS
• 17.3 Counters we added have the correct values - PASS
• 17.4 Add -8 to then get a default counter on each node - PASS
• 17.5 Add 5 to then get a default counter on each node - PASS
• 17.6 Get then add 5 to a default counter on each node - PASS
• 17.7 Counters we added have the correct values - PASS
• 17.8 Distributed Set get - PASS
• 17.9 Distributed Set size - PASS
• 17.10 Distributed Set add() - PASS
• 17.11 Distributed Set addAll() - PASS
• 17.12 Distributed Set contains() - PASS
• 17.13 Distributed Set containsAll() - PASS
• 17.14 Distributed Set remove() - PASS
• 17.15 Distributed Set removeAll() - PASS
• 17.16 Distributed Set addAll() - PASS
• 17.17 Distributed Set clear() - PASS
• 17.18 Distributed Set addAll() - PASS
• 17.19 Distributed Set retain() - PASS
• 17.20 Partitioned Transactional maps put - PASS
• 17.21 Partitioned Transactional maps get - PASS
• 17.22 Get the value of a new value - PASS
• 17.23 Atomic Value set() - PASS
• 17.24 Get the value after set() - PASS
• 17.25 Atomic Value compareAndSet() - PASS
• 17.26 Get the value after compareAndSet() - PASS
• 17.27 Atomic Value getAndSet() - PASS
• 17.28 Get the value after getAndSet() - PASS
• 17.29 Atomic Value destory() - PASS
• 17.30 Get the value after destroy() - PASS
• 17.31 Work Queue add() - PASS
• 17.32 Check the work queue stats - PASS
• 17.33 Work Queue addMultiple() - PASS
• 17.34 Check the work queue stats - PASS
• 17.35 Work Queue takeAndComplete() 1 - PASS
• 17.36 Check the work queue stats - PASS
• 17.37 Work Queue takeAndComplete() 2 - PASS
• 17.38 Check the work queue stats - PASS
• 17.39 Work Queue destroy() - PASS
• 17.40 Check the work queue stats - PASS

Case 6: Scale the number of nodes in the ONOS cluster - PASS

• 6.1 Checking ONOS Logs for errors - No Result
• 6.2 Scaling from 3 to 5 nodes - No Result
• 6.3 Start new nodes - PASS
• 6.4 Set up ONOS secure SSH - PASS
• 6.5 Checking ONOS service - PASS
• 6.6 Checking ONOS nodes - PASS

Case 8: Compare ONOS Topology view to Mininet topology - PASS

Compare topology objects between Mininet and ONOS

• 8.1 Comparing ONOS topology to MN topology - PASS
• 8.2 Hosts view is consistent across all ONOS nodes - PASS
• 8.3 Hosts information is correct - PASS
• 8.4 Host attachment points to the network - PASS
• 8.5 Clusters view is consistent across all ONOS nodes - PASS
• 8.6 There is only one SCC - PASS
• 8.7 Device information is correct - PASS
• 8.8 Links are correct - PASS
• 8.9 Hosts are correct - PASS
• 8.10 Checking ONOS nodes - PASS

Case 7: Running ONOS Constant State Tests - PASS

• 7.1 Check that each switch has a master - PASS
• 7.2 Read device roles from ONOS - PASS
• 7.3 Check for consistency in roles from each controller - PASS
• 7.4 Get the intents from each controller - PASS
• 7.5 Check for consistency in Intents from each controller - PASS
• 7.6 Compare current intents with intents before the scaling - PASS
• 7.7 Get the OF Table entries and compare to before component scaling - PASS
• 7.8 Leadership Election is still functional - PASS

Case 4: Verify connectivity by sending traffic across Intents - PASS

Ping across added host intents to check functionality and check the state of the intent

• 4.1 Check Intent state - PASS
• 4.2 Ping across added host intents - PASS
• 4.3 Check leadership of topics - PASS
• 4.4 Wait a minute then ping again - PASS

Case 15: Check that Leadership Election is still functional - PASS

• 15.1 Run for election on each node - PASS
• 15.2 Check that each node shows the same leader and candidates - PASS
• 15.3 Find current leader and withdraw - PASS
• 15.4 Check that a new node was elected leader - PASS
• 15.5 Check that that new leader was the candidate of old leader - PASS
• 15.6 Run for election on old leader( just so everyone is in the hat ) - PASS
• 15.7 Check that oldLeader is a candidate, and leader if only 1 node - No Result

Case 17: Check for basic functionality with distributed primitives - PASS

Test the methods of the distributed primitives (counters and sets) throught the cli

• 17.1 Increment then get a default counter on each node - PASS
• 17.2 Get then Increment a default counter on each node - PASS
• 17.3 Counters we added have the correct values - PASS
• 17.4 Add -8 to then get a default counter on each node - PASS
• 17.5 Add 5 to then get a default counter on each node - PASS
• 17.6 Get then add 5 to a default counter on each node - PASS
• 17.7 Counters we added have the correct values - PASS
• 17.8 Distributed Set get - PASS
• 17.9 Distributed Set size - PASS
• 17.10 Distributed Set add() - PASS
• 17.11 Distributed Set addAll() - PASS
• 17.12 Distributed Set contains() - PASS
• 17.13 Distributed Set containsAll() - PASS
• 17.14 Distributed Set remove() - PASS
• 17.15 Distributed Set removeAll() - PASS
• 17.16 Distributed Set addAll() - PASS
• 17.17 Distributed Set clear() - PASS
• 17.18 Distributed Set addAll() - PASS
• 17.19 Distributed Set retain() - PASS
• 17.20 Partitioned Transactional maps put - PASS
• 17.21 Partitioned Transactional maps get - PASS
• 17.22 Get the value of a new value - PASS
• 17.23 Atomic Value set() - PASS
• 17.24 Get the value after set() - PASS
• 17.25 Atomic Value compareAndSet() - PASS
• 17.26 Get the value after compareAndSet() - PASS
• 17.27 Atomic Value getAndSet() - PASS
• 17.28 Get the value after getAndSet() - PASS
• 17.29 Atomic Value destory() - PASS
• 17.30 Get the value after destroy() - PASS
• 17.31 Work Queue add() - PASS
• 17.32 Check the work queue stats - PASS
• 17.33 Work Queue addMultiple() - PASS
• 17.34 Check the work queue stats - PASS
• 17.35 Work Queue takeAndComplete() 1 - PASS
• 17.36 Check the work queue stats - PASS
• 17.37 Work Queue takeAndComplete() 2 - PASS
• 17.38 Check the work queue stats - PASS
• 17.39 Work Queue destroy() - PASS
• 17.40 Check the work queue stats - PASS

Case 6: Scale the number of nodes in the ONOS cluster - FAIL