Docs: Configure Spinnaker's Usage of Redis

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker relies on its cache for a few key purposes:

Session data for logins
Bake logs and history
Recording which triggers have already been acted upon when using docker/jenkins triggers
ETC.

Most of these can be configured to your needs, whether it is to make Spinnaker more responsive, or reduce the load on a downstream dependency. Some of these settings apply only to SQL or REDIS depending upon configuration. The below are some examples of adjusting these configurations.

Configure infrastructure caching

By default, Spinnaker’s Clouddriver service spawns “caching agents” every 30 seconds. These agents are scheduled and coordinated across all instances of Clouddriver, allowing you to reduce the load on any single instance of Clouddriver by creating more replicas of the service.

To adjust how this caching happens, Clouddriver exposes a few properties that can be set in ~/spinnaker-kustomize/overlays/config/files/clouddriver-local.yml:

# How many seconds (default 30s) between runs of agent. Lowering this number
# means the resources in the Spinnaker UI will be updated more frequently,
# at the cost higher API/quota usage of your cloud provider.
sql.poll.intervalSeconds:


# How many seconds (default 300s, 5 minutes) Clouddriver will wait to reschedule
# an agent that never completes (never throws an error or returns cache data).
# If your agents are taking a long time to complete their cache cycles
# successfully and Clouddriver is prematurely rescheduling them, you can try to
# raise this number.
sql.poll.timeoutSeconds:

Configure pipeline cleanup

If you see that your Redis memory usage is growing unbounded, it’s probably because the orchestration engine stores all pipeline executions forever by default. This can be configured by setting the following properties in ~/spinnaker-kustomize/overlays/config/files/orca-local.yml:

pollers:
  oldPipelineCleanup:
    enabled: true                  # This enables old pipeline execution cleanup (default: false)
    intervalMs: 3600000            # How many milliseconds between pipeline cleanup runs (default: 1hr or 3600000)
    thresholdDays: 30              # How old a pipeline execution must be to be deleted (default: 30)
    minimumPipelineExecutions: 5   # How many executions to keep around (default: 5)

tasks:
  controller:
    daysOfExecutionHistory: 30      # How many days to keep old task executions around.

There are MANY more configuration options available as needed. See the Properties files in spinnaker for the various services for information and ask in slack for help.

Docs: Externalize Redis

Mon, 01 Jan 0001 00:00:00 +0000

One of the easiest ways to improve Spinnaker’s reliability at scale is to use an external Redis. The default redis component isn’t configured to be production-ready. If you have a hosted Redis alternative, or a database team managing a Redis installation, we highly recommend using that.

Configure a Spinnaker-wide Redis

First, determine the URL of your Redis installation. Some examples include:

redis://some.redis.url:6379: Redis running at some.redis.url on port 6379.
redis://admin:passw0rd@some.redis.url:6379: Same as above, but with a username/password pair.
redis://admin:passw0rd@some.redis.url:6379/1: Same as above, but using database 1. See SELECT documentation .

We will refer to this as $REDIS_ENDPOINT. You can add this to a common configuration for all services ~/spinnaker-kustomize/base/spinnaker.yml

services:
  redis:
    baseUrl: redis://some.redis.url:6379
    enabled: true
    host: 0.0.0.0
    port: 6379

Configure per-service Redis

If your single Redis node is overloaded (unlikely - spinnaker mostly uses SQL at this point), you can configure Spinnaker’s services to use different Redis endpoints. You can adjust the redis baseUrl in each service with an overridden configuration like so:

services.redis.baseUrl: $REDIS_ENDPOINT

Using a hosted Redis

Gate requires keyspace notifications to be enabled in Redis, and tries to configure this when it starts up. Some hosted Redis services disable the CONFIG command, blocking Gate from modifying the configuration. In this case:

Manually set the configuration parameter notify-keyspace-events to gxE on your Redis instance by following the documentation provided by your hosted Redis provider.
Disable automatic Redis configuration in Gate by adding the following to your gate-local.yml file:

  redis:
    configuration:
      secure: true