Page History

...

The diagram below shows the overall architecture of the system.

Image Modified

• 1. The External Application registers for KafkaService. It makes POST call with the application name.  
  2. The response could contain server Ip, port etc, needed for connectivity along with the consumer group id generated for this app. 
  3. The External Application makes a POST REST call to the Kafka Integration App.

...

• 1. This time to subscribe for an event.
     The contents of the REST call could be - App name, Event Type and the allocated consumer group-id.
  2. If there is no listener for this event type the Event Manager module within the App will register a listener for the specific event with ONOS. 
     If there is a listener already, the event manager will not create a listener.
  3. The response for the POST will be 200 ok instead of 201 created. This is because the resource is created on an
     external entity

...

• 1. . The response could contain

...

• 1. the Kafka Topic information or just a plain message.
  2. The external app would then use this information to connect to the Kafka server and register itself as a consumer for the topic it had registered earlier.
  3. At some later point in time we receive our first event from ONOS. The Event Manager module will pick up the event. 
     It will convert the data format to the common GPB format and Marshall the data using the GPB library. The Data is then passed along to the Kafka Manager. 
  4. The Kafka Manager module will publish the event to the topic. 
  5. The external app will receive the event in GPB defined message format.

...

REST APIs 

1. POST (register) - This will be used by non native apps to register for the Kafka service.
2. POST (subscribe) - This will be used by non native apps to subscribe for a sepcific a new event type.
3. GET - Return the list of event types supported by the Kafka Integration App
4. DELETE (subscribe) - This will be used by non native apps to deregister for a specific event type.
5. DELETE (register) - This will be used by non native apps to deregister for a kafka service.

CONFIGURATION

The Application loads a json config file at start up. This config file will have Kafka config information.

...

1. There will be one topic per event type. Each external app will need  to provide be given a unique consumer groupId. 
2. Event subscription by external apps is a two step process - They must first register and then subscribe to a specific event call. 

3. As a first step we will only export Device Events  and Link Events to consumers and worry about Packet Ins and Packet Out later. 

4. Once the framework is in place it should be relatively easy to add support for other event types. 
5. In the scenario where the external app loses connectivity with the Kafka server, and does not come back up within the retention period (the time duration for which Kafka will retain messages)  the onus is on the non native app to rebuild its current view of the network state via existing ONOS REST APIs. 

FAQ

1. Is it possible that before Kafka Integration App tries to create a topic, the external app can try to subscribe to that topic? 
   Yes, this is possible. In such a scenario Kafka will automatically create the topic. This is similar to the case where the
   producer tries to publish a message to a non existent topic. Kafka will create a topic automatically.
   
   
2. What is the purpose of having a Group Id and App name info in the POST REST call? 
   GroupId is necessary to make sure there is no conflict among multiple external apps. When the external apps tries to consume data
   each app should be in a separate consumer group. This way the message is delivered to all the external apps from a single topic. The
   App name is primarily for keeping track of which apps have registered. May be we could have a GET REST call to show the current registered apps.
   
   
3. How does the external app know what event types are supported ?
   A GET REST API will be provided to show the list of events supported
   
   
4. How does the external app know how many partitions were created for the topic?
   This I am thinking will be sent as part of the response to the POST call. The external
   app needs this information, so it can spin up appropriate number of threads. 
   (Num of Consumer Threads = Num of Partitions)
   
   
5. What is the topic name that gets created?
   The topic name is the same as event type.

...