Monday, April 16, 2018

Queues - RabbitMQ vs SQS

RabbitMQ
RabbitMQ supports powerful message routing via exchange. This is very important when we need to run the same job on a specific server, group of servers or all servers. Our application sends one message and exchange will route it. RabbitMQ also has vhosts so that multiple applications can share the same RabbitMQ server but be isolated from each other (we can create unique logins for separate applications to access their vhosts). RabbitMQ can be setup in clusters for redundancy / failover and will acknowledge receipt of messages.
RabbitMQ has a powerful GUI which is accessible http://localhost:15672/. Hosting for RabbitMQ offers fewer choices and is more expensive.

AWS SQS
AWS SQS takes care of managing our queues. AWS SDK offers low level access but shoryuken is a higher level integration (shoryuken author acknowledges sidekiq as inspiration). Retrying failed jobs will happen automatically unless the messages is explicitly deleted. We can only delay jobs for 15 minutes (or we get The maximum allowed delay is 15 minutes (RuntimeError)). Recurring jobs are not supported but there are workarounds with AWS lambda and CloudWatch. AWS SQS UI is decent and we can use AWS SDK to access data directly. SQS has other interesting features such as long polling, batch operations and dead letter queues. SQS also has FIFO queues which guarantee he order in which messages are sent and received (and does not allow dupes). However, FIFO queues only allow 300 TPS (much less that regular SQS). Shoryuken works with standard and FIFO queues.
Hosting - easy to setup and cheap (pay for what you use) but obviously only available on AWS. SQS is a great choice when we need to run LOTS of jobs or when we do not care about more advanced options such as scheduling.


RabbitMQ
Amazon SQS
DescriptionRabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
Pros• It's fast and it works with good metrics/monitoring
• Ease of configuration
• I like the admin interface
• Easy to set-up and start with
• Intuitive work through python
• Standard protocols
• Durable
• Written primarily in Erlang
• Simply superb
• Completeness of messaging patterns
• Easy to use, reliable
• Low cost
• Simple
• Doesn't need to maintain
• Delayed delivery upto 12 hours
• Has a max message size (currently 256K)
• Delayed delivery upto 15 mins only
Cons• Needs Erlang runtime. Need ops good with Erlang runtime
• Too complicated cluster/HA config and management
• Configuration must be done first, not by your code
• SQS has guaranteed delivery, but messages can be delivered more than once.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.