<img src="https://jenkins.onosproject.org/view/QA/job/postjob-BM/lastSuccessfulBuild/artifact/VPLSfailsafe_onos-2.2_20-builds_graph.jpg", alt="VPLSfailsafe", style="width:525px;height:350px;border:0">

commit 4d73add23e18f5353e2b17d232bed1cf6912beb4 (HEAD -] onos-2.2, origin/onos-2.2)
Author: Charles Chan [rascov@gmail.com]
AuthorDate: Fri Feb 14 13:23:57 2020 -0800
Commit: Charles Chan [rascov@gmail.com]
CommitDate: Fri Jun 19 17:36:04 2020 -0700

Phased recovery

Case 1: Starting up 3 node(s) ONOS cluster - PASS

Set up ONOS with 3 node(s) ONOS cluster

• 1.1 Constructing test variables - PASS
• 1.2 Apply cell to environment - PASS
• 1.3 Uninstalling Atomix - PASS
• 1.4 Uninstalling ONOS package - PASS
• 1.5 Creating ONOS package - PASS
• 1.6 Installing Atomix - PASS
• 1.7 Installing ONOS package - PASS
• 1.8 Set up ONOS secure SSH - PASS
• 1.9 Checking ONOS service - PASS
• 1.10 Starting ONOS CLI sessions - PASS
• 1.11 Checking ONOS nodes - PASS
• 1.12 Checking ONOS applications - PASS
• 1.13 Starting Mininet - PASS
• 1.14 Activate apps defined in the params file - No Result
• 1.15 Set ONOS configurations - No Result
• 1.16 App Ids check - PASS

Case 2: - PASS

• 2.1 Discover hosts using pings - No Result
• 2.2 Load VPLS configurations - PASS
• 2.3 Check VPLS configurations - PASS
• 2.4 Check interface configurations - PASS
• 2.5 Check network configurations for vpls application - PASS
• 2.6 Check vpls app configurations - PASS
• 2.7 Check connectivity - PASS
• 2.8 Loading vpls configuration in case any configuration was missed. - PASS

Case 50: Check connectivity before running all other tests. - PASS

• 50.1 Check intent states - PASS
• 50.2 Testing connectivity... - PASS

Case 100: Bring down one host at a time and test connectivity. - FAIL

• 100.1 Kill link between s1 and h1. - PASS
• 100.2 Check intent states - FAIL
• Not all Intents in installed state
• 100.3 Testing connectivity... - PASS
• 100.4 Re-adding link between s1 and h1. - PASS
• 100.5 Discover h1 using ping. - No Result
• 100.6 Check intent states - PASS
• 100.7 Testing connectivity... - PASS
• 100.8 Kill link between s2 and h2. - PASS
• 100.9 Check intent states - FAIL
• Not all Intents in installed state
• 100.10 Testing connectivity... - PASS
• 100.11 Re-adding link between s2 and h2. - PASS
• 100.12 Discover h2 using ping. - No Result
• 100.13 Check intent states - PASS
• 100.14 Testing connectivity... - PASS
• 100.15 Kill link between s3 and h3. - PASS
• 100.16 Check intent states - FAIL
• Not all Intents in installed state
• 100.17 Testing connectivity... - PASS
• 100.18 Re-adding link between s3 and h3. - PASS
• 100.19 Discover h3 using ping. - No Result
• 100.20 Check intent states - PASS
• 100.21 Testing connectivity... - PASS
• 100.22 Kill link between s4 and h4. - PASS
• 100.23 Check intent states - FAIL
• Not all Intents in installed state
• 100.24 Testing connectivity... - PASS
• 100.25 Re-adding link between s4 and h4. - PASS
• 100.26 Discover h4 using ping. - No Result
• 100.27 Check intent states - PASS
• 100.28 Testing connectivity... - PASS
• 100.29 Kill link between s5 and h5. - PASS
• 100.30 Check intent states - FAIL
• Not all Intents in installed state
• 100.31 Testing connectivity... - PASS
• 100.32 Re-adding link between s5 and h5. - PASS
• 100.33 Discover h5 using ping. - No Result
• 100.34 Check intent states - PASS
• 100.35 Testing connectivity... - PASS
• 100.36 Kill link between s6 and h6. - PASS
• 100.37 Check intent states - FAIL
• Not all Intents in installed state
• 100.38 Testing connectivity... - PASS
• 100.39 Re-adding link between s6 and h6. - PASS
• 100.40 Discover h6 using ping. - No Result
• 100.41 Check intent states - PASS
• 100.42 Testing connectivity... - PASS

Case 200: Bring down one switch at a time and test connectivity. - FAIL

• 200.1 Delete s5. - PASS
• 200.2 Check intent states - FAIL
• Not all Intents in installed state
• 200.3 Testing connectivity... - PASS
• 200.4 Add s5. - PASS
• 200.5 Reconnecting links on s5. - No Result
• 200.6 Check intent states - PASS
• 200.7 Testing connectivity... - FAIL
• Connectivity is NOT as expected.

Case 300: Stop one ONOS node at a time and test connectivity. - FAIL

• 300.1 Stop ONOS node 0. - PASS
• 300.2 Testing connectivity... - FAIL
• Connectivity failed.
• 300.3 Restart ONOS node 0 and checking status of restart. - PASS
• 300.4 Checking ONOS nodes - FAIL
• Apps are NOT the same across all nodes.
• 300.5 Check intent states - PASS
• 300.6 Testing connectivity... - FAIL
• Connectivity is NOT as expected.

Case 310: Kill one ONOS node at a time and test connectivity. - FAIL

• 310.1 Killing ONOS node 1. - PASS
• 310.2 Waiting for ONOS to restart. - No Result
• 310.3 Check intent states - PASS
• 310.4 Testing connectivity... - FAIL
• Connectivity is NOT as expected.

Case 400: Bring down one link at a time and test connectivity. - FAIL

• 400.1 Bring down link: s1 to h1 - PASS
• 400.2 Check intent states - PASS
• 400.3 Testing connectivity... - FAIL
• Connectivity failed.
• 400.4 Adding link: s1 to h1. - PASS
• 400.5 Check intent states - PASS
• 400.6 Testing connectivity... - FAIL
• Connectivity is NOT as expected.