changestream-to-redis

This program listens to a MongoDB Change Stream, and publishes changes to Redis. It is designed to work with the cultofcoders:redis-oplog Meteor package and meant as an alternative to to oplogtoredis with support for protectAgainstRaceConditions: false.

Setup

1. Install redis-oplog with the following configuration:
   • externalRedisPublisher: true.
   • globalRedisPrefix: "${database}.", e.g., globalRedisPrefix: "meteor.".
2. Deploy changestream-to-redis with the following environmental variables:
   • (required) MONGO_URL, e.g., mongodb://localhost:27017/meteor.
   • (required) REDIS_URL, e.g., redis://localhost:6379/1.
   • (optional) DEBUG.
     • If set, all events are logged before being sent to Redis.
   • (optional) DEDUPLICATION, e.g., 120.
     • If set, all events are deduplicated on Redis. That allows you to deploy multiple instances of oplogtoredis listening to the same MongoDB database and pushing to the same Redis database.
   • (optional) FULL_DOCUMENT.
     • If not set, only IDs will be sent to Redis, i.e., it will behave just like oplogtoredis.
     • If set, it has to be one of the values accepted by MongoDB (required, updateLookup, or whenAvailable), and you can configure your collections to use protectAgainstRaceConditions: false.
   • (optional) FULL_DOCUMENT_COLLECTIONS, e.g., notifications,users.
     • If not set, there will be one change stream, fetching full documents from all collections, according to the FULL_DOCUMENT flag.
     • If set, there will be two change streams. First, listening to the configured collections, fetching full documents when available (i.e., inserts) and according to the FULL_DOCUMENT flag. Second will listen to other collections, fetching only their IDs.

Limitations

• No change stream resumption. It is planned, but at the moment the program is entirely stateless.
• No error handling. As soon as the change stream or Redis communication fails, the program exits. It is planned, though changestream-to-redis is meant to restart as soon as it exits.
• No monitoring. There is no monitoring of any kind, but both a healt-checking endpoint and Prometheus metrics are planned.

Development

This is a standard Rust application: cargo fmt will format the code, cargo clippy will check it, and cargo test will run tests.