Page History

...

Depending on the requirements of a service, how the contents of a store is distributed between nodes can have different characteristics (e.g., strongly consistent, eventually consistent, ...). This is made possible by having each service's store implement the appropriate distribution mechanism. Currently, store for Mastership service uses Hazelcast's distributed structures as a strongly consistent backend, and store for Device, Link, and Host services uses peer to peer messaging, implemented on Netty, to replicate states and background optimistic replication technique complemented by a background gossip protocol to ensure eventual consistency.

Simply put, the same subsystems of two different nodes synchronize directly with one another through the Store. The Store only synchronizes the states of the subsystem that it is part of; A DeviceStore, for example, cannot be used to synchronize Host informationdoes not have any knowledge of host information is tracked. The figure below summarizes this for two nodes and a subsystem "A" that is part of both.

...

At the time of this writing, all services with the exception of topology management have access to distributed stores. The distributed version of topology store just simply relies on the distributed versions of device, link and host stores.

Event ordering

For the eventually consistent stores, events are partially ordered with an approach similar to vector clocks. The logical clock used by the DeviceStore and LinkStore is a combination of the number of mastership handoffs for a device since its discovery (its term number), and a sequence number local to the node that is incremented per timestamp generationobserved event. The HostStore relies on system time (the wall clock), due to the lifespan of Host objects and their mobility that prevents them from being tied to a particular device. 

Cluster Management

...