Add a new Redis instance
GitLab can make use of multiple Redis instances. These instances are functionally partitioned so that, for example, we can store CI trace chunks from one Redis instance while storing sessions in another.
From time to time we might want to add a new Redis instance. Typically this will be a functional partition split from one of the existing instances such as the cache or shared state. This document describes an approach for adding a new Redis instance that handles existing data, based on prior examples:
- Dedicated Redis instance for Trace Chunk storage.
- Create dedicated Redis instance for Rate Limiting data.
This document does not cover the operational side of preparing and configuring the new Redis instance in detail, but the example epics do contain information on previous approaches to this.
Step 1: Support configuring the new instance
Before we can switch any features to using the new instance, we have to support configuring it and referring to it in the codebase. We must support the main installation types:
- Source installs (including development environments) - example MR
- Omnibus - example MR
- Helm charts - example MR
In the application code, we need to define a fallback instance in case the new instance is not configured. For example, if a GitLab instance has already configured a separate shared state Redis, and we are partitioning data from the shared state Redis, our new instance's configuration should default to that of the shared state Redis when it's not present. Otherwise we could break instances that don't configure the new Redis instance as soon as it's available.
You can define a
Gitlab::Redis::Wrapper (the base class for all Redis instances)
that defines the instance to be used if this one is not configured. If we were
Foo instance that should fall back to
SharedState, we can do that
module Gitlab module Redis class Foo < ::Gitlab::Redis::Wrapper # The data we store on Foo used to be stored on SharedState. def self.config_fallback SharedState end end end end
We should also add specs like those in
to ensure that this fallback works correctly.
Step 2: Support writing to and reading from the new instance
When migrating to the new instance, we must account for cases where data is either on:
- The 'old' (original) instance.
- The new one that we have just added support for.
As a result we may need to support reading from and writing to both instances, depending on some condition.
The exact condition to use varies depending on the data to be migrated. For the trace chunks case above, there was already a database column indicating where the data was stored (as there are other storage options than Redis).
This step may not apply if the data has a very short lifetime (a few minutes at most) and is not critical. In that case, we may decide that it is OK to incur a small amount of data loss and switch over through configuration only.
If there is not a more natural way to mark where the data is stored, using a feature flag may be convenient:
- It does not require an application restart to take effect.
- It applies to all application instances (Sidekiq, API, web, etc.) at the same time.
- It supports incremental rollout - ideally by actor (project, group, user, etc.) - so that we can monitor for errors and roll back easily.
Step 3: Migrate the data
We then need to configure the new instance for GitLab.com's production and staging environments. Hopefully it will be possible to test this change effectively on staging, to at least make sure that basic usage continues to work.
After that is done, we can roll out the change to production. Ideally this would be in an incremental fashion, following the standard incremental rollout documentation for feature flags.
When we have been using the new instance 100% of the time in production for a while and there are no issues, we can proceed.
Proposed solution: Migrate data by using MultiStore with the fallback strategy
We need a way to migrate users to a new Redis store without causing any inconveniences from UX perspective. We also want the ability to fall back to the "old" Redis instance if something goes wrong with the new instance.
- No downtime.
- No loss of stored data until the TTL for storing data expires.
- Partial rollout using Feature Flags or ENV vars or combinations of both.
- Monitoring of the switch.
- Prometheus metrics in place.
- Easy rollback without downtime in case the new instance or logic does not behave as expected.
It is somewhat similar to the zero-downtime DB table rename. We need to write data into both Redis instances (old + new). We read from the new instance, but we need to fall back to the old instance when pre-fetching from the new dedicated Redis instance that failed. We need to log any issues or exceptions with a new instance, but still fall back to the old instance.
The proposed migration strategy is to implement and use the MultiStore. We used this approach with adding new dedicated Redis instance for session keys. Also MultiStore comes with corresponding specs.
The MultiStore looks like a
redis-rb ::Redis instance.
module Gitlab module Redis class Foo < ::Gitlab::Redis::Wrapper ... def self.redis # Don't use multistore if redis.foo configuration is not provided return super if config_fallback? primary_store = ::Redis.new(params) secondary_store = ::Redis.new(config_fallback.params) MultiStore.new(primary_store, secondary_store, store_name) end end end end
MultiStore is initialized by providing the new Redis instance as a primary store, and old (fallback-instance) as a secondary store.
The third argument is
store_name which is used for logs, metrics and feature flag names, in case we use MultiStore implementation for different Redis stores at the same time.
By default, the MultiStore reads and writes only from the default Redis store.
The default Redis store is
secondary_store (the old fallback-instance).
This allows us to introduce MultiStore without changing the default behavior.
MultiStore uses two feature flags to control the actual migration:
For example, if our new Redis instance is called
Gitlab::Redis::Foo, we can create two feature flags by executing:
bin/feature-flag use_primary_and_secondary_stores_for_foo bin/feature-flag use_primary_store_as_default_for_foo
use_primary_and_secondary_stores_for_foo feature flag, our
Gitlab::Redis::Foo will use
MultiStore to write to both new Redis instance
and the old (fallback-instance).
If we fail to fetch data from the new instance, we will fallback and read from the old Redis instance.
We can monitor logs for
Gitlab::Redis::MultiStore::ReadFromPrimaryError, and also the Prometheus counter
For pipelined commands (
multi), we execute the entire operation in both stores and then compare the results. If they differ, we emit a
Gitlab::Redis::MultiStore:PipelinedDiffError error, and track it in the
gitlab_redis_multi_store_pipelined_diff_error_total Prometheus counter.
Once we stop seeing those errors, this means that we are no longer relying on the data stored on the old Redis store. At this point, we are probably safe to move the traffic to the new Redis store.
use_primary_store_as_default_for_foo feature flag, the
MultiStore will use
primary_store (new instance) as default Redis store.
Once this feature flag is enabled, we can disable
use_primary_and_secondary_stores_for_foo feature flag.
This will allow the MultiStore to read and write only from the primary Redis store (new store), moving all the traffic to the new Redis store.
Once we have moved all our traffic to the primary store, our data migration is complete. We can safely remove the MultiStore implementation and continue to use newly introduced Redis store instance.
MultiStore implements read and write Redis commands separately.
NOTE: The Ruby block passed to these commands will be executed twice, once per each store. Thus, excluding the Redis operations performed, the block should be idempotent.
When a command outside of the supported list is used,
method_missing will pass it to the old Redis instance and keep track of it.
This ensures that anything unexpected behaves like it would before.
gitlab_redis_multi_store_method_missing_total counter and
a developer will need to add an implementation for missing Redis commands before proceeding with the migration.
||Value not found on the Redis primary store. Read from the Redis secondary store successful.|
||Pipelined command executed on both stores successfully but results differ between them.|
||Method missing. Falling back to execute method on the Redis secondary store.|
||Prometheus Counter||command, instance_name||Client side Redis MultiStore reading fallback total|
||Prometheus Counter||command, instance_name||Redis MultiStore pipelined command diff between stores|
||Prometheus Counter||command, instance_name||Client side Redis MultiStore method missing total|
Step 4: clean up after the migration
We may choose to keep the migration paths or remove them, depending on whether or not we expect self-managed instances to perform this migration. gitlab-com/gl-infra/scalability#1131 contains a discussion on this topic for the trace chunks feature flag. It may be - as in that case - that we decide that the maintenance costs of supporting the migration code are higher than the benefits of allowing self-managed instances to perform this migration seamlessly, if we expect self-managed instances to cope without this functional partition.
If we decide to keep the migration code:
- We should document the migration steps.
- If we used a feature flag, we should ensure it's an ops type feature flag, as these are long-lived flags.
Otherwise, we can remove the flags and conclude the project.