A few quick tips to get started contributing to OnosSystemTest:

...

• Subscribe to the System Testing mailing list <onos-test@onosproject.org>. All mailing lists can be found here.

...

• Sign up for an account on onosproject.org

...

• to be able to commit test codes to gerrit.onosproject.org.

...

• If you are not familiar with Gerrit workflow,

...

• check out this tutorial: Gerrit Workflow for System Test Development.

...

• Check out our guides about the TestON

...

• framework: System Testing Guide.

Fundamentals on

...

Authoring a

...

Test Case

Because we are targeting all tests to be automated in an CI enva Continuous Integration (CI) environment, there are several fundamental principles that we have to adhere to when writing test cases and driver files:

1. Portability - allowing allow the test case to be run in other similar test environments; at the same time easier for us to merge the community contributed test in , as well as merging community contributed tests into our production environment;.
2. Stability - the test should take in considerations of must handle various environment (e.g. response time of starting onos ONOS on VMs vs. on Bare metalsMetals), therefore, ; test case should must have stability in passing or failing results;.
3. Clarity - we should try all efforts strive to make the Python cases easy to understand and follow;simple to comprehend.
4. Debugging Debuggability - use logging capability capabilities liberally and to catch all cases of exceptions, and make sure that to ensure test failures don't go silently.

In order to achieve the these objectives, it is essential to have a good understanding of the Jenkins - TestON interaction interactions and abstraction abstractions in running a test case. The following diagram illustrates the interaction and abstraction of TestON and Jenkins.

TestON Scripting General Guidelines:

Due to the many components required for system tests, it is possible that one tester's environment can be drastically different from another's.  In ONLAB, we have a production test environment to run the current published test suites. In time, we expect the community to contribute to improving the test suites, or creating new test suites that will also be run on this env. We set up a set of test development guidelines to achieve the following objectives:

1. Portability: 

• testers should be able to clone the tests; set up system environment variables accordingly; and run the tests without test script modifications;

• contributors should be able to commit changes and new test suites to be run on ONLAB production testbed without test script modifications.

2. Stability:

• robustness - i.e. minimize script-causing test failures.

3. Clarity:

• tests should produce useful onos failure information for further debug;

• reports should provide a glance of onos health from each suite;

• Standardize driver & tests documentation.

Note: Due to the high fluidity of current onos features and APIs, OnosSystemTest version will track with onos version.

...

TestON Scripting General Guidelines:

• Pre-requisites

...

• and testbed environment setting - should be set manually, and/or with CI framework

...

• (i.e.

...

• through Jenkins jobs):

  ...

  • • First Time Setup:
    • 1. TestStations:
         • should have a root account of "

  ...

  • • 1. • sdn" (password: rocks) to run TestON cli (so all hosts in the test infrastructure has the same sdn/rocks credential)
         • should be able to

  ...

  • • 1. • login to all nodes specified in the cell. See this guide to setup ssh keys.

  ...

  • • 1. "ONOS Bench" and cells:

  ...

  • • 1.  set up per "ONOS from Scratch".

  ...

  • • 1. Mininet (OCN) host: set up

  ...

  • • 1. per  "Mininet Walkthrough".
         

  ...

  • • 1. Note: it is possible to run "TestStation", "onos Bench" and "Mininet" on the same host, with .topo file set up accordingly.
    • General Steps:
      1. Clean up TestON and

  ...

  • • 1. Mininet before each test run.
      2. Set OnosSystemTest version, and update OnosSystemTest by typing: "git pull

  ...

  • • 1. OnosSystemTest" from gerrit.onosproject.org.
      2. Set

  ...

  • • 1. ONOS version, and update ONOS by typing: "git pull onos" from gerrit.onosproject.org.
      2. Compile/

  ...

  • • 1. Build ONOS.
      2. Set up ONOS Java Virtual Machine (JVM)

  ...

  • • 1. related configurations if the default configuration is not desirable.
      2. Run test cases

  ...

  • • 1. .  See scripting guidelines below.

  ...

  • • Optional

  ...

  • • :
      1. Run post-test tasks, such as

  ...

  • • 1. :
         • Data storage
         • Result publishing
      2. Teardown by uninstalling ONOS and cleaning up Mininet

  ...

  • • 1. .
         
         
  • OnosSystemTest/TestON Script should perform in:
    
    • Test suite naming, see Test Plans.
    • README file
      • Should describe the test topology to run the default test case.
      • Should explain the main goal of the test.
      • Provide any additional requirements needed to run the test.
    • testname.params
      • Use

  ...

  • • • environment variable names to reference to components - no static IP's.
      • Move any hardcoded values to this file - e.g. sleep times, counters, file names... etc.
      • If any modifications need to be made to the test, it should be done in this file.
    • testname.topo
      • Use

  ...

  • • • environment variable names to reference to components - no static IP's

  ...

  • • • .
        
      • Leave out any passwords for the login information - Password-less login should be setup.
    • testname.py
      • Log

  ...

  • • • ONOS version/commit.
      • Set prompt, terminal type.

  ...

  • • • Handle test case dependencies.
      • Explicit

  ...

  • • • ONOS cell creation and setting.
      • Explicit activation of bundles/apps.
      • Test case specific app configurations for non-default values - cell apps should be specified in the params file.
      • Test dependencies, Mininet topologies, and helper functions should be stored in the Dependency

  ...

  • • • directory located in the test

  ...

  • • • directory.
      • Test log should log the relevant config information - Log is cheap; be as verbose as possible to help the debugging process.
      • Avoid static reference to paths, files - put any static references in the .params file.
      • Check and log summary of onos exceptions, errors, warnings - Ideally, this should

  ...

  • • • be done after each test case/
      • Handling of Test results - write to test log and

  ...

  • • • /tmp file for post-test processing purposes - Try to assert on every test result so that it can be shown on the wiki.