How to Use Redis for Caching and Pub/Sub in Python | by Lynn Kwong

Picture from Unsplash.

Redis is an open-source in-memory knowledge construction retailer that can be utilized as an in-memory key-value database, caching system, and pub/sub message dealer. Redis is particular in that each one its knowledge is saved in reminiscence quite than on disk, which makes it extraordinarily quick and a well-liked choice for caching.

In a previous article, the necessities of Redis knowledge sorts and customary instructions have been launched. On this article, we are going to concentrate on easy methods to use Redis for caching and Writer/Subscriber (pub/sub) in Python.

You possibly can set up a Redis server instantly in your laptop. Nonetheless, for studying functions, it’s advisable to make use of a Docker container for Redis as a result of you’ll be able to all the time use the newest model of Redis for testing. To begin a Redis server with a Docker container, run the next instructions in a shell.

A excessive port 16379 is used to keep away from potential port conflicts in your machine. Moreover, a Docker community is created to make it simpler to attach the Redis server with redis-cli:

Now you can run the widespread instructions with redis-cli as demonstrated within the previous article. Nonetheless, the main target of this text is easy methods to use Redis in Python, so let’s proceed.

To entry the Redis server in Python, you want a Python Redis consumer. The advisable one is redis-py, which is mature and properly supported. It’s at present seen as “the best way to go” for Python.

To make use of redis-py, we must always set up it on our laptop. It is strongly recommended to put in it in a virtual environment so it gained’t mess up with present libraries. For simplicity, we are going to use conda to create a digital atmosphere:

The (redis) on the command line signifies that the digital atmosphere is efficiently created and activated. In case you need to study extra about conda, this article could be a good reference.

Then to put in redis-py, run:

$ pip set up redis

Observe that the library to put in is redis, not redis-py. To make it simpler to run Python instructions interactively, we may also set up iPython within the digital atmosphere:

$ pip set up ipython

We are able to then begin iPython and work together with Redis:

In case your Redis server has authentication enabled, you’ll be able to specify the password with the password parameter. You too can specify the db parameter in case your Redis server has a number of databases. The default database quantity is 0, which is the default one to make use of whenever you don’t specify one.

Observe that it’s best to specify the port accurately, right here it’s the customized excessive port 16379, not the classical 6379. Additionally word that the keys and values for Redis can solely be bytes, strings, integers, and float numbers. For finest follow, we must always usually solely use significant strings because the keys. Nonetheless, for a worth with a non-basic kind like an object or array, it have to be first transformed to a string or bytes earlier than it may be set as a worth in Redis. The json.dumps perform is often used to transform dictionaries or lists in Python to JSON strings.

Then again, as we see, by default, all responses are returned as bytes in Python. If we need to decode all string responses from a Redis consumer, we are able to specify the decode_responses and encoding parameters once we create the consumer:

In case you are unfamiliar with the ideas of string encoding/decoding and Unicode/UTF-8, this article may be useful.

To delete a Redis key, simply use the delete technique of the consumer:

Within the outcome, 1 means the hot button is deleted efficiently, and 0 signifies that the important thing doesn’t exist.

When we now have many Redis keys to verify, there’s a handy technique referred to asscan_iter, which has similarities to the SCAN command however is far more handy to make use of as a result of we don’t want to recollect and specify the cursor:

Right here the colon (:) is simply used to separate the thing kind and the id and truly doesn’t have any particular that means. The scan_iter technique expects a Redis pattern as an enter. If no sample is specified, all of the keys can be returned. Particularly, the star “*” matches any variety of any characters. It’s equal to the “.*” sample in a daily expression.

Above we now have launched the very elementary utilization of Redis in Python. Most Redis strategies in Python have their native redis-cli command counterparts. In case you are considering or want to make use of different strategies, you’ll be able to check with the previous article that focuses on native redis-cli instructions.

Now let’s concentrate on the Redis pipeline and pub/sub which aren’t generally used with redis-cli, however extra generally in a driver (Python on this submit).

For Redis, the pipeline is a option to execute a number of instructions on the identical time. The instructions can be buffered and just one request can be despatched to the Reids server. On this approach, the overhead for speaking between the Redis server and consumer is lowered dramatically and the velocity/effectivity may be elevated. The pipeline could be very useful when it is advisable run many Redis instructions sequentially.

The pipeline is just like the transaction in SQL databases. Nonetheless, as an alternative of first beginning a transaction after which committing, right here you first create a pipeline object after which execute:

As we see, the pipeline could be very easy to make use of. It has extra advanced settings as properly which makes it work in additional advanced conditions.

Redis is a generally used light-weight message/queue or publish/subscribe (pub/sub) system. For instance, Airflow makes use of it because the dealer that forwards messages from scheduler to employee.

As with different message/queue methods, we have to create a channel and subscription earlier than we are able to publish and obtain messages. The channel is typically referred to as topic in some message/queue methods such because the Pub/Sub service of Google Cloud Platform.

Truly, we don’t have to create a channel explicitly. The channel is created mechanically when it’s subscribed to for the primary time, as proven above. Furthermore, we are able to subscribe to a number of channels on the identical time. We are able to even subscribe to a number of channels by sample, as we are going to see quickly with the psubscribe technique under.

Let’s now publish some messages to the channel:

It ought to be famous that the Redis consumer (redis_cli) is used to publish messages to a channel, not the PubSub object created above. The quantity returned is the variety of subscribers to which the message was delivered. Let’s create a brand new subscription and see if the quantity adjustments accordingly.

On this instancepsubscribe means a subscription can be created by sample. The created subscription will hearken to all of the channels matching the desired sample.

As we see, once we publish to channel-*, the return is 1 that means the message was solely revealed to 1 subscriber, particularly the one created with sample channel-*. Nonetheless, when the message was revealed to channel-1 once more, the return worth adjustments to 2, that means the message was revealed to 2 subscribers. Truly, there is just one PubSub object created and used. Nonetheless, this PubSub object has two subscriptions/subscribers, one to channel channel-1 and the opposite one to all of the channels matching channel-*.

Let’s now get the revealed messages from the system. To do that, we have to name the get_message technique of the PubSub object.

The get_message technique will get the following message if one is on the market, in any other case None. It returns a dictionary with 4 keys:

  • kind — The kind of the info. The worth may be subscribe, psubscribe, message, and so on. message means the precise knowledge that’s revealed, the opposite ones imply the subscription sorts and may be seen as metadata. We’re usually solely within the knowledge with the message kind.
  • sample — The sample for the channels. It’s None for all messages besides the pmessage kind, as proven above. On this instance, it implies that the message is revealed to channels matching the sample channel-*.
  • channel — The channel to which the message is revealed.
  • knowledge — The precise message knowledge revealed. For the “metadata” subscriptions, it’s the variety of subscriptions for the PubSub object on the time when the get_message technique was referred to as. For the message kind, it’s the precise message that was despatched.

In case you are curious as me, a “metadata” message is revealed when a brand new subscription is created and when it’s unsubscribed.

Usually you gained’t hassle an excessive amount of with the metadata and solely have to work with the actual messages which are revealed.

As we are able to see from the straightforward examples above, it is rather easy to create a channel (subject) and subscription after which publish/obtain messages with Redis. You possibly can then create some logic to parse the info obtained and use it in your utility.

On this article, the most typical use circumstances of Redis in Python are launched. Along with the previous article on utilizing native Redis instructions, it’s best to have a reasonably good understanding of Redis now and may begin to use it in your work.

More Posts