Primary node can help with coordination of multiple node deployed in a cloud. One of the projects I’ve worked on had a NodeJS worker that ran multiple types of tasks. I wanted to upgrade this setup in order to be easily scalable, have a primary node (or coordinator) that triggers the tasks and continue processing even when some of the nodes fail.
The Checklist
- All nodes may participate in an “election” to choose the coordinator
- Support any number of nodes, including 1 node setup
- Handle node fail (even the coordinator node) without impacting the flow
- Allow new nodes to join the party
- Don’t depend on an expensive technology (paid or resource hungry)
Besides the main scope of the solution I also needed to ensure that the election of the coordinator follows these 3 basic principles:
- Termination: the election process must complete in a finite time
- Uniqueness: only one node can be coordinator
- Agreement: all other nodes know who’s the coordinator.
After I established that I have all the scenarios in mind, I started investigating different algorithms and solutions that the market uses (Apache Zookeeper, Port Locking, Ring Networks). However, most of these require a lot of setup or were incompatible in a multi server setup and I also wanted to embrace a KISS approach so continue reading to see the solution.
The Primary Node Election Algorithm
- Node generates a random numeric id
- Node retrieves a COORDINATOR_ID key from Redis
- If key !NULL
- We have a coordinator
- Wait Z minutes (e.g. Z = 1 hour)
- GoTo Step2
- If key NULL
- No coordinator announce
- Push the id from Step1 in a Redis list
- Waits X seconds (depending on how long the deployment takes, e.g. 10 seconds)
- Retrieve all items in the list and extract the highest number
- If result === node id
- Current node is primary
- Set Redis key COORDINATOR_ID with expiry Z+X
- Do all the hard work 🙂
- Wait Z minutes
- GoTo Step2
Downside of this solution is that if the coordinator node fails, it actually takes 2*Z until a new election takes place.
There’s room for improvement so please don’t hesitate to leave a feedback 🙂