Page History

...

The Application loads a json config file at start up. This config file will have Kafka config information.

CLUSTERING SUPPORT

1. We will be using a Distributed Work Queue(DWQ) as the primitive.
2. There will be Leadership elections for two separate topics – WORK_QUEUE_PUBLISHER, WORK_QUEUE_CONSUMER.
3. The Leader for WORK_QUEUE_PUBLISHER will publish events to DWQ.
4. The Leader for WORK_QUEUE_CONSUMER will consume events from DWQ and send it to KafkaServer.
5. In case of failure of one of the Leaders and the Leader has failed to mark the task consumed from DWQ as complete, the task(in our case event) will be returned to the DWQ. The new Leader will pick this up from DWQ.
6. There is a rare possibility that the Leader for publisher failed before it can queue the event to DWQ. This will result in a loss of event.
7. Since there is only one consumer from DWQ FIFO ordering of events is guaranteed.

...