Introduction to Redis Streams

Redis has introduced a new type of data structure in version 5.0.0, a significant data structure for building stream-processing services and message queues like RabitMQ and Apache Kafka it gives you the most simple and powerful to startup.

Redis Streams are primarily an append-only data structure like writing a log file and unlike the nature of Redis pub/sub, it cares a lot more about data loss by using consumer and consumer group patterns and you shouldn't confused up they are two different things.

Redis pub/sub and Redis Streams

let’s have a short review on Redis pub/sub and learn the differences and benefits we get when we work with the streams. Redis incorporated the publish-subscribe pattern in version 2.0.0, which allows clients to subscribe to one or more channels and receive messages as long as they are connected. the publisher produces messages in a channel without knowledge of what (if any) subscribers there may be. Subscribers express interest in one or more and only receive messages that are of interest and subscribed to, without knowledge of what (if any) publishers there are. there is no data safety in Redis pub/sub and a case of a network connection failure between a publisher and subscriber messages are gone be lost and by the time of reconnecting the messages in-between are lost forever and there is no acknowledgment system in order the consumer to ack the publisher which it had or hadn’t received the message. Actually, we could use lists or hashes to build a message queue that has data persistency and handle the data loss in Redis pub/sub pattern but its not the very right tool to do that and we miss a lot of amazing features because of the nature of pub/sub itself in Redis, so starting with version 5.0.0 you can use Streams, a new datatype implemented with a log data structure and a set of very flexible commands. in Redis streams, unlike pub/sub the order matters to us and keeps track of the message orders and the sequences without having to lock the database to write new data. also, it includes the concept of Consumer Groups and it help us to scale our consumers to not having any bottleneck. so let's wrap it up these differences:

1. Redis pub/sub not handling’s the failures and data loss on the other hand thanks to the consumer and publisher pattern alongside the acknowledgment system that already exists in Redis Streams we never lose a message or task anymore.
2. Redis Streams will keep the data like a log file with sequence and others until we decided to clear it.
3. Redis Streams are append-only and we don’t need to lock the database for writing new data and we have some cool features like the time we write the data without the need to use any other things by just using auto-generated Id that is built-in.
4. Redis Streams have the concept of consumer groups and acknowledgment and it help us to scale easily and have more fail tolerance by adding more workers and consumers.
5. on the other hand, you should see Redis pub/sub more like protocol that can move your message around.

Streams basics

Conceptually, a Stream in Redis is a list where you can append entries. Each entry has a unique ID and a value and a field name. the field and value must be a string. the ID is auto-generated by default and it includes a timestamp but it also could be custom with some pattern and rules to noticed.

Redis Streams commnads

• XACK: removes one or multiple messages from the pending entries list (PEL) of a stream consumer group.
• XADD: Appends the specified stream-entry to the stream at the specified key.
• XAUTOCLAIM: transfers ownership of pending stream entries that match the specified criteria.
• XCLAIM: changes the ownership of a pending message.
• XDEL: Removes the specified entries from a stream.
• XGROUP: manage the consumer groups associated with a stream data structure.
• XINFO: retrieve different information about the streams and associated consumer groups.
• XLEN: Returns the number of entries inside a stream.
• XPENDING: inspect the list of pending messages.
• XRANGE: returns the stream entries matching a given range of IDs.
• XREAD: Read data from one or multiple streams.
• XREADGROUP: a special version of the XREAD command with support for consumer groups.
• XREVRANGE: returning the entries in reverse order.
• XTRIM: trims the stream by evicting older entries.

XADD

add a new entry to a stream by giving the stream names and values. this command returns an Entry ID that is composed of two parts, part one includes milliseconds is the local time in the Redis node which generating the streams ID and sequenceNumber is used for entries created in the same millisecond. the field and value must be a string and you can replace “*” with costume and it returns the ID for example:

redis> XADD mystream * name Sara surname OConnor
"1616882756284-0"
redis> XADD mystream * field1 value1 field2 value2 field3 value3
"1616882756285-0"

where and how to use Redis streams

so this is the part that things going to be heavy and I want to talk about Software architects. you should first get to know what is an event streaming system design looks like there is a well-explained video from the IBM team that says some introduction to Kafka by the way, Apache Kafka is a framework implementation of a software bus using stream-processing. it's really similar to Redis streams and they have the same concept but you should remind that Kafka is the complete tool but Redis Stream is just a data type and it gives you the core to build a broker, consumer, or producer.

so as you can see Redis streams give you the core and basics to build a broker or message queues or any event-driven system but actually it's better not to see it like a message queue as they have some differences. we have shared concepts like ack and groups and topics that they are the streams in Redis.

here is an example of using Redis stream in node.js:

'use strict';

var redis = require('redis');
var client1 = redis.createClient();
var client2 = redis.createClient();
var client3 = redis.createClient();

client1.xadd('mystream', '*', 'field1', 'm1', function (err) {
    if (err) {
        return console.error(err);
    }
    client1.xgroup('CREATE', 'mystream', 'mygroup', '$', function (err) {
        if (err) {
            return console.error(err);
        }
    });

    client2.xreadgroup('GROUP', 'mygroup', 'consumer', 'Block', 1000, 'NOACK',
        'STREAMS', 'mystream', '>', function (err, stream) {
            if (err) {
                return console.error(err);
            }
            console.log('client2 ' + stream);
        });

    client3.xreadgroup('GROUP', 'mygroup', 'consumer', 'Block', 1000, 'NOACK',
        'STREAMS', 'mystream', '>', function (err, stream) {
            if (err) {
                return console.error(err);
            }
            console.log('client3 ' + stream);
        });


    client1.xadd('mystream', '*', 'field1', 'm2', function (err) {
        if (err) {
            return console.error(err);
        }
    });

    client1.xadd('mystream', '*', 'field1', 'm3', function (err) {
        if (err) {
            return console.error(err);
        }
    });

});