In order to make administration of an ONOS cluster easier, ONOS comes with a small set of tools that the administrator can use to interact with ONOS cluster from a remote platform, which can be either their laptop or some designated administration host.
Downloading & Installing Admin Tools
The admin tools are available for download as a compressed tar from Maven central or from Google drive. The
onos-admin-<version>.tar.gz file can be unrolled at a desired location on the machine(s) from which the ONOS cluster will be remotely administered. The PATH environment variable should be set to include the ONOS admin tools directory, e.g:
The admin tools primarily operate either using the ONOS CLI or ONOS REST API. The clients for these are the native ssh command and curl command, respectively. The ONOS CLI (ssh to port
8101) access is secured using the
onos-user-key command, and the ONOS REST API (curl to port
8181) access is secured using
onos-user-password command. Both of these commands are part of the standard ONOS distribution and are located under the top-level
bin directory, e.g.
/opt/onos/bin. See the following section on securing the ONOS cluster.
To simplify the remote administration it is recommended to capture the details about the ONOS cluster being managed by creating a file that will hold definitions of the ONOS cluster instances:
Save this file under a name that will make it easy to remember which ONOS deployment cluster it refers to.
Then to set the environment to point to that ONOS pod, simply source in the file, e.g.:
After this, and after exporting the
PATH environment variable as indicated above, you should be able to type in the following commands to manage the cluster, e.g.:
The above are just a few examples. Any of the ONOS admin commands that are intended to command a specific instance take the numeric argument or IP address as the first argument. These include
Securing ONOS Cluster
To configure passwordless CLI access, the operator must run
onos-user-key tool as follows from each machine in the cluster. This tool is available under the ONOS
Similarly, it is recommended to change the default username and password for the REST API using
onos-user-password tool on each ONOS cluster node as follows:
By configuring the ONOS cluster in this manner, not only it will become more secure, it will also become more convenient to manage using automated tools without having to enter credentials each time.
Documentation & Example Usage
The following sections provides a quick overview of the individual administrative tools and their usage.
The onos-diagnostics tool collects various information from the running ONOS cluster and packages it into one, easy-to-share archive file. This tool is distributed as part of the ONOS software itself (under bin directory), but is also available as part of a small archive of remote tools to administer an ONOS cluster (onos-admin-*.tar.gz).
In order to run the onos-diagnostics tool, the machine/account from which the tool runs must be allowed to remotely connect to the ONOS CLI. This is accomplished by registering the user’s public RSA/DSA key with each ONOS instance. To make this easier another tool onos-user-key has been provided as part of the base ONOS distribution to modify the ONOS configuration appropriately to make this possible and, equally important, to make the ONOS deployment secure.
Since the tools contacts all ONOS node cluster instances, it needs to know the IP addresses of those machines. To avoid having to specify these IP addresses as part of the command, you can export the ONOS_INSTANCES environment variable to specify the addresses as a space-separated list. Here’s an example of how to set the variable:
The tool also accesses the ONOS REST API to collect logs and for this it requires the REST API username and password credentials. These credentials can be provided either via ONOS_WEB_USER and ONOS_WEB_PASSWD environment variables or via command options (see usage below)
Once enabled, the onos-diagnostics tool can be run as follows:
There is an option that allows for naming the resulting archive file for differentiation between different cluster instances, e.g.
By default onos-diagnostics will use ONOS_PROFILE to collect the diags. We use the profiles to customize the behavior of the tool.
The resulting /tmp/*-diags.tar.gz file will contain all relevant information about the ONOS cluster.
The following is the usage help for the onos-diagnostics tool:
Alternatively, it is possible to use onos-diagnostics-k8s in Kubernetes enabled environments. The tool will produce the same results of onos-diagnostics and relies only on kubectl commands.
The tools needs to know the names of the pods. To avoid having to specify these names as part of the command, you can export the ONOS_PODS environment variable to specify the names as a space-separated list. Here’s an example of how to set the variable:
The tool needs to know the Karaf home (path from the mount point). To avoid having to specify this path as part of the command, you can export the KARAF_HOME environment variable:
Once enabled, the onos-diagnostics-k8s tool can be run as follows:
The following is the usage help for the onos-diagnostics-k8s tool:
Since ONOS Apache Karaf log can be spread across multiple files, i.e.
karaf.log.3, etc. and since each ONOS cluster node has its own set of such logs, it can be cumbersome to find relevant information in the logs purely manual means. The
onos-log-query tool has been provided to make log stitching, filtering and collating much easier. The tool can operate on standalong log files, but it works well with information collected via the
The tool is designed to splice together the different
karaf.log* files and then to optionally to narrow them to include only the specified date/time interval. This is done using the
-t options. Since the splicing and filtering operation may take some time, the tool stores the processed output into a separate file, which by default is called
query.log. This name can be changed using the
-n option. This allows the user to keep results of multiple different queries and it also allows other types of processing to be applied on these files using other external tools without repeatedly incurring the cost of stitching and filtering a potentially very large data-sets.
The simplest way to run the tool is from the root of the directory produced by unrolling the ONOS diagnostics archive, which is where the node-level directories are. For example, to stitch and filter the logs to only focus on the specific duration of output, you could do the following:
This will produce stitched and filtered versions of the log for each individual node, as well as a collated version containing merge of all node logs. These files will be called
anomaly.log; there will be one for each of the nodes in the respective node directory and also the collated one at the top-level directory.
The -f flag specifies the start of the time period (inclusive) and the -t flag specified the end of the tim period (also inclusive). If either is not specified, the logs will be included from the beginning or until the end. The
-x option indicates that only timestamped log entries should be included, which means stack-traces and other multi-line log entries will be reduced to just the time-stamped line of the log. The -n option allows you to provide a custom name under which your results should be saved; if not specified, the logs will be stored in
To facilitate remote management of ONOS applications the onos-app tool has been provided as a way to easily upload new application binaries via REST API. Of course, applications can also be listed, uninstalled, activated and deactivated using the CLI, but installation of apps via CLI only allows installing apps via a URL to application binaries located on a web-server elsewhere. The onos-app tools allows uploading application OAR file from the administration machine directly.
The following is the usage:
For example, to upload a super duper ONOS app, whose binaries are located locally in a file
~/onos/apps/super-duper-6.28.oar file, to the ONOS cluster through one of its instances at
10.45.32.69, you would use the following command:
! postfix in the
reinstall! commands indicates that the application should also be activated immediately after the installation. Otherwise, the commands mean what their names clearly imply.
ONOS supports dynamic YANG compilation via GUI and via REST API. To make this even more convenient, the onos-compile-yang command has been provided. Internally, it uses the REST API, but should be far easier to use from the remote Unix shell.
The command allows uploading and compiling an individual
.yang file, or a collection of yang files located in a given
.tar.gz file or even just a plain local directory. YANG compilation will result in generation of Java classes from the included YANG models, compilation of those classes into an ONOS application
.oar bundle and automatic installation and activation of the application. The newly registered YANG models can be immediately listed via YANG REST API, GUI or CLI commands. The appllication will derive it's name and identifier from the name of the uploaded file or directory. It can be seen among the list of applications installed and active on the running ONOS cluster.
For example, to upload, compile and register
some-funky-model.yang file, one could do the following:
Notice that the ONOS cluster does not need to be rebuilt, stopped or reconfigured in any way. Once can simply dynamically add and remove YANG models at will.
The onos command is simply a convenience wrapper that allows the administrator to securely open the ssh session to any of the ONOS cluster nodes. As with all the admin commands, one can use either the DNS name, IP address or a numeric index of the ONOS node - provided the OC# environment variables have been properly setup. The following are examples of usage: