How to Contribute to System Test

A few quick tips to get started contributing to OnosSystemTest:

1) Subscribe "onos-test: mailing list: Mailing Lists;

2) Sign up for an account on onosproject.org so as to be able to commit test codes to gerrit.onosproject.org.

TestON Scripting General Guidelines:

Due to the many components required for system tests, it is possible that one tester's environment can be drastically different from another's.  In ONLAB, we have a production test environment to run the current published test suites. In time, we expect the community to contribute to improving the test suites, or creating new test suites that will also be run on this env. We set up a set of test development guidelines to achieve the following objectives:

1. Portability: 

• testers should be able to clone the tests; set up system environment variables accordingly; and run the tests without test script modifications;

• contributors should be able to commit changes and new test suites to be run on ONLAB production testbed without test script modifications.

2. Stability:

• robustness - i.e. minimize script-causing test failures.

3. Clarity:

• tests should produce useful onos failure information for further debug;

• reports should provide a glance of onos health from each suite;

• Standardize driver & tests documentation.

Note: Due to the high fluidity of  current onos features and APIs, OnosSystemTest version will track with onos version.

Guidelines:

1. Pre-requisites  and testbed environment setting - should be set manually, and/or with CI framework, e.g. through Jenkins jobs:

1. 1.  (First time setup only)TestStations:
      1. should have a root account of "admin" to run TestON cli;
      2. should be able to logon using env variable to localhost, OC[1~7], OCI using password-less login;
      3. "admin" should be able to sudo without password.
   2. (First time setup only)"onos Bench" and cells: set up per "ONOS from Scratch"
   3. (First time setup only)Mininet (OCN) host: set up per "Mininet Walkthrough".
      Note: it is possible to run "TestStation", "onos Bench" and "Mininet" on the same host, with .topo file set up accordingly.
   4. Clean up TestON and MN before each test run;
   5. Set OnosSystemTest version, and git pull "OnosSystemTest" from gerrit.onosproject,org;
   6. Set onos version, and git pull onos from gerrit.onosproject.org;
   7. Compile/build onos;
   8. Setup onos JVM related configurations if default is not desirable;
   9. Run test cases - see Scripting guidelines below;
   10. (Optional) Run post-test tasks, such as data storage, result publishing, etc.
   11. (Optional)Teardown: onos uninstall and mn cleanup.
       
       
2. OnosSystemTest/TestON Script should perform in:
   
   1. Test suite naming, see Test Plans.
   2. README file
      1. should describe the test topology to run the default test case.
   3. testname.params
      1. Use env variable names to reference to components - no static IP's;
   4. testname.topo
      1. Use env variable names to reference to components - no static IP's;
         
   5. testname.py
      1. Log onos version/ commit
      2. Set prompt, terminal type
      3. Handel test case dependencies
      4. Explicit onos cell creation and setting;
      5. Explicit activation of bundles/apps;
      6. Test case specific app configurations for non-default values;
      7. Test dependencies should be stored in "tests/testname/dependencies" folder, for example, a mininet topo file; and managed at test run time;
         
      8. Test log should log the relevant config information;
      9. Test logics;
      10. Avoid static reference to paths, files;
      11. Check and log summary of onos exceptions, errors, warnings;
      12. Handling of Test results - write to test log and  /tmp file for post-test processing purposes;