The test results graphs that show up on the Test Results Wiki Page are created using the R programming language, immediately following the conclusion of a scheduled test.
CHO graphs are refreshed every hour during a CHO test, and the events, failures, and exceptions are recorded and posted to the wiki.
Each test result (besides SCPF and PLATdockertest) is illustrated as a trend graph, displaying the results of the last 20 builds of the specific test.
The number of test cases executed is equal to the sum of passed and failed test cases. Therefore, tests may contain skipped test cases if the executed test cases (sum of passed and failed cases) do not equal the number of planned test cases.
The trend graphs for SCPF display the result of a specific value over 50 builds.
Each SCPF test contains detailed results about the variables measured during the test, which is illustrated using stacked and/or error bar graphs. Depending on the test, stacked bar graphs display latencies or throughput values, with the value at the top of the stack being the sum of all of the bars. Error bar charts display the top half of the error bars.
Similar to the trend graphs of individual tests, these graphs display the results of all tests that are fully passing vs. tests that contain at least one error and the number of tests run for the corresponding build number.
For each test category, the overall pie charts display percentage of passed test cases vs. percentage of failed test cases, and number of executed test cases vs. number of skipped test cases.
The build statistics graph displays a summary of all test suites as stacked bar graphs, with the stacks containing the percent of passed, failed, and skipped test cases for each test category.
Each script is divided into three major steps:
Each graph present on the wiki may be customized. All graphs may have the branch, name, and build number modified. However, each script may have specific variables that may also be changed:
The total number of builds to show may be set. The default setting is 20.
The number of builds to show may be set, as well as whether to show old flow results (supported for branch onos-1.12 or newer).
The tests to include, total number of builds to show, and the title of the graph may be set.
The tests to include and the build to show may be set, as well as whether to show passed/failed test cases, or executed/skipped test cases.
The title of the graph, build to show, and tests to display may be set.