Introducing Starsky, an opinionated, high-level NodeJS module for building services with RabbitMQ. This post is full of nerdy stuff involving programming, message queues and why we decided to build our own RabbitMQ module. If you want to cut to the chase, go checkout out the project and get started now!
At Recurly, we've been busy releasing new features and improvements for our growing customer base at an increasing pace. As Recurly continues to grow so also grows:
the quantity of internal services
the need to coordinate communication between those services
the need for organized job processing
the need for deeper insight into application level events
A message queue is an ideal fit for this type of architecture. For our needs, RabbitMQ supports the requirements of our use cases, has proven to scale well in production environments, and has paid support available in the event of an emergency.
After we made the decision that RabbitMQ would be a good fit for a number of tasks, we found ourselves repeatedly asking the following questions:
What exchange types will I need?
When should I declare a message as persistent?
Which queues should be durable?
How to create baseline preferred options across various projects?
What are the necessary options for high-availability?
We decided that the last two questions were most important to us and figured that by answering them, everything else should fall into place. This led us to the following set of conventions:
A single topic exchange
All messages are persistent
All messages are acked or nacked
All queues are durable
Naming conventions drive mirror policies
With this base set of needs in mind we evaluated the open source tools available and found Hutch, a Ruby gem from GoCardless. Hutch quite elegantly implemented everything we were looking for so we set out to find a NodeJS module to compliment it. There were a few low level RabbitMQ NodeJS modules available at the time but nothing that fit quite so nicely as Hutch so we decided to write our own, and with a play on words, named it Starsky. Starsky is modeled directly after the elegant approach taken by Hutch and it's baked in opinions are the same:
Durable queues & exchanges
Using Starsky in combination with Hutch, we now have a solution to consistently apply our preferred RabbitMQ approach across the organization. Starsky is open sourced on GitHub. Feel free to use the project, read the source and contribute! https://github.com/recurly/starsky