The Owl release (1.14) features a new architecture which physically decouples cluster management, service discovery, and persistent data storage from the ONOS nodes themselves. These functions are now the responsibility of a separate Atomix cluster. The new ONOS/Atomix architecture, however, introduces several breaking changes to the way clusters are configured and managed and requires additional setup and configuration steps. This guide will explain how to configure Atomix and ONOS in production.

In order to understand how the cluster is configured in ONOS 1.14, it's important first to understand how it is architected. In past versions, ONOS embedded Atomix nodes to form Raft clusters, replicate state, and coordinate state changes. In ONOS 1.14, that functionality is moved into a separate Atomix cluster:

Prior to bootstrapping the ONOS cluster, an Atomix cluster must first be formed for data storage and coordination. ONOS nodes are then configured with a list of Atomix nodes to which to connect. The Atomix cluster can be formed by either configuring the official Atomix Docker image or by downloading and running the Atomix distribution.

Atomix Configuration

Atomix can be configured using either JSON or HOCON configuration files. The only requirement for the Atomix configuration is that it include a Raft partition group which can be used by ONOS for data storage and coordination.

cluster {
  name: atomix
  discovery {
    type: bootstrap
    nodes.1 {
      id: atomix-1
      address: "10.192.19.111:5679"
    }
    nodes.2 {
      id: atomix-2
      address: "10.192.19.112:5679"
    }
    nodes.3 {
      id: atomix-3
      address: "10.192.19.113:5679"
    }
  }
}

management-group {
  type: raft
  partitions: 1
  storage.level: disk
  members: [atomix-1, atomix-2, atomix-3]
}

partition-groups.raft {
  type: raft
  partitions: 7
  storage.level: disk
  members: [atomix-1, atomix-2, atomix-3]
}

In the above configuration, Atomix is configured to use a single Raft partition for system management, and seven Raft partitions for data storage. The partition-groups section configures the partitions that will be used by ONOS directly. The Atomix distribution includes an example ONOS configuration in the /examples directory. For more information on Atomix configuration and the concepts of partitions and partition groups, consult the Atomix documentation.

ONOS Configuration

The second part of the ONOS cluster configuration is the ONOS configuration itself. In past releases, the cluster has been configured either using cluster.json or by running the onos-form-cluster script. The Owl release preserves cluster.json but no longer supports onos-form-cluster. However, the format of cluster.json has changed to accommodate changes in communication patterns within the ONOS and Atomix clusters. In past releases, ONOS nodes formed a cluster by communicating with one another to elect leaders using the Raft protocol. As a consensus protocol, Raft required strict cluster membership information to preserve consistency while forming the cluster by counting votes. But since the Raft protocol is moved to Atomix in ONOS 1.14, the ONOS cluster itself no longer has any such strict requirement for cluster membership. In other words, an ONOS node can now discover its peers using dynamic discovery mechanisms, and the ONOS cluster as a whole can tolerate the failure of all but one node. This is done by discovering ONOS nodes through the Atomix cluster:

When an ONOS node is bootstrapped, it connects to the external Atomix cluster for data storage and coordination. Upon connecting, the ONOS node notifies Atomix of its existence and location. Atomix then broadcasts the new node's information to all other connected ONOS nodes, and connected ONOS nodes subsequently connect directly back to the new ONOS node for peer-to-peer communication. This allows ONOS nodes to dynamically discover one another while still preserving the same performance characteristics as in past releases.

But the introduction of service discovery for the ONOS controller changes its cluster configuration. In past releases, cluster.json was used to list each ONOS node's peers and the distribution of Raft partitions in the cluster. But ONOS nodes now have no knowledge of their peers or the available storage partitions. Instead, ONOS nodes need only be configured to locate and connect to Atomix to discover each other and store state.

To facilitate the new cluster architecture, the format of cluster.json is changed to list the set of Atomix nodes rather than the set of ONOS nodes. This is done by listing the nodes in the storage field:

{
  "name": "onos",
  "storage": [
    {
      "id": "atomix-1",
      "ip": "10.192.19.111",
      "port": 5679
    },
    {
      "id": "atomix-2",
      "ip": "10.192.19.112",
      "port": 5679
    },
    {
      "id": "atomix-3",
      "ip": "10.192.19.113",
      "port": 5679
    }
  ]
}

The format is the same as the format for the nodes field in past releases, but the listed IP/port combinations are for Atomix nodes rather than ONOS nodes. In past releases, ONOS nodes could be assigned human readable identifiers. In order to facilitate this in the Owl release, we added an additional node field to describe the local node.

{
  "name": "onos",
  "node": {
    "id": "onos-1",
    "ip": "10.192.19.121",
    "port": 9876
  },
  "storage": [
    {
      "id": "atomix-1",
      "ip": "10.192.19.111",
      "port": 5679
    },
    {
      "id": "atomix-2",
      "ip": "10.192.19.112",
      "port": 5679
    },
    {
      "id": "atomix-3",
      "ip": "10.192.19.113",
      "port": 5679
    }
  ]
}