Which Messaging System to Choose? Kafka, Valkey, RabbitMQ, NATS... Oh, So Many!

When building highly communicative distributed systems, one common headache is choosing the right messaging system. There are so many options available: Kafka, Valkey (born from Redis), RabbitMQ, and NATS. Each has its pros and cons. So, in this post, I’ll jot down some notes to compare them for myself, figuring out which one suits which task.

Kafka

What is it?

Kafka is a massive data streaming platform designed to handle huge volumes of real-time data. Think of it as a large data pipeline that’s fast, durable, and easily scalable.

When should you use it?

• For real-time data analytics: When you need to process continuously flowing data to get immediate results.
• For centralized log collection: Gathering logs from multiple services into one place.
• For Event Sourcing: Using every event in the system as the primary data source.

What I like

• Very high throughput and low latency.
• Scalable and fault-tolerant.
• Durable message storage.

What I don’t like as much

• Installation and maintenance can be quite complex.
• Consumes more system resources than others.
• Not ideal for simple queueing tasks where workers just pick up one job and finish.

Valkey (Redis)

What is it?

Valkey, also known as Redis (Valkey is a community-driven fork), is an incredibly fast in-memory database. People often use it as a cache or a simple message broker, utilizing data structures like lists for queues.

When should you use it?

• For basic queues: Tasks that don’t require advanced features, just a simple queue.
• For Pub/Sub: Suitable for chat applications or real-time notifications.
• For Caching: This is its specialty.

What I like

• Extremely fast due to in-memory operation.
• Easy to use and install, not complex.
• Versatile with various data structures available.

What I don’t like as much

• If not configured properly, data can be lost during a power outage (not durable by default).
• Not suitable for storing very large messages.
• Not designed for heavy data streaming or complex message routing.

RabbitMQ

What is it?

RabbitMQ is a mature and robust message broker, primarily using the AMQP protocol. It’s known for its comprehensive messaging capabilities, flexible routing, and reliability.

When should you use it?

• For asynchronous tasks: Offloading heavy tasks to be processed in the background, so users don’t have to wait.
• For work queues: When you have a pile of tasks and want multiple workers to process them collaboratively.
• For conditional message delivery: When you want to send messages to specific consumers based on various conditions.

What I like

• Rich in features, a veteran in the field.
• Extremely flexible message routing.
• Supports various messaging patterns (fanout, direct, topic).

What I don’t like as much

• May not match Kafka’s throughput.
• Requires careful configuration for high availability.
• Messages are typically read and then deleted; not designed for long-term storage.

NATS

What is it?

NATS is a messaging system built for the Cloud-Native, IoT, and Microservices era. Its key strengths are high speed and being extremely lightweight, focusing on simplicity.

When should you use it?

• For Microservice communication: Fast and reliable communication between services.
• For IoT device data: Suitable for receiving a massive number of small messages from various devices.
• For command and control systems: Sending commands to control distributed systems.

What I like

• Truly fast and lightweight.
• Easy to install and manage.
• Supports both Pub/Sub and Request/Reply.

What I don’t like as much

• No built-in message persistence (requires NATS Streaming/JetStream).
• Routing features are not as sophisticated as RabbitMQ.
• Not designed for long-term message storage like Kafka.

So, which one should you choose?

Ultimately, the choice depends on what you want to achieve:

• If you need a massive, durable, and scalable data pipeline: Go with Kafka. You won’t be disappointed.
• If you need fast, simple queues or want to do Pub/Sub or Caching: Valkey (Redis) is the answer.
• If you need a mature message broker with complex and reliable routing: It has to be RabbitMQ.
• If you need a fast, lightweight messaging system suitable for Microservices: NATS is a very interesting option.

Before making your decision, ask yourself what kind of throughput your task requires, how long you need to store data, whether message delivery is complex, and how much effort you can put into maintaining it.