Spinnaker – Runbooks

Docs: API Rate Limiting

Mon, 01 Jan 0001 00:00:00 +0000

If your Spinnaker deployment has API clients interacting with it, enabling and knowing how to operate the API rate limiter can help keep Spinnaker reliable through spikes of heavy traffic or rogue clients.

Intro

The API Rate Limiter currently supports limiting individual authenticated and anonymous principals, bucketing requests into windows that are refreshed every interval as well as global & per-principal learning mode. Both the capacity, as well as the window size, are configurable globally and per-principal.

Some metrics are generated by the rate limiter:

rateLimit.throttling: Counter. Number of enforced throttled requests.
rateLimit.throttlingLearning: Counter. Number of un-enforced throttled requests.
rateLimit.principal.throttled: Counter. A tagged metric recording the number of throttled requests per-principal. Tag key is principal.
rateLimit.principal.remaining: Gauge. A tagged metric recording the number of remaining requests per-principal. Tag key is principal.

Combined with controller.invocations, these metrics can give you a good sense of your on-going rate limiting configuration.

In the case of failures (Redis unavailable) or configuration errors (principal included in both enforcing and ignoring lists), a request will gracefully fallback to learning mode.

Enabling in gate.yml

The rate limiter requires Redis to be available for Gate.

rateLimit:
  enabled: true
  learning: true
  redis:
    enabled: true
  # The rate (in seconds) that the bucket will be replenished
  rateSeconds: 10
  # The number of allowed requests in the window of rateSeconds
  capacity: 500
  # Specific principals can be given different capacities
  capacityByPrincipal:
  - principal: anonymous
    override: 1000
  # Similarly, rateSeconds can be overridden per-principal
  rateSecondsByPrincipal:
  - principal: anonymous
    override: 5
  # A list of principals that are being enforced. Handy for cases where you want
  # to incrementally enable the rate limiter
  enforcing:
  - example@example.com
  # A list of principals that are in learning mode. This can be useful if you
  # want to give some principals unlimited power. Reconsider doing this :)
  ignoring:
  - unmetered-example@example.com

Integrating clients

The rate limiter exposes four HTTP Headers on each request that a client can ingest to play nicely with the rate limiter:

X-RateLimit-Capacity: Int. Total capacity assigned to the principal.
X-RateLimit-Remaining: Int. Total number of requests left before reset.
X-RateLimit-Reset: Date. When the bucket will be replenished.
X-RateLimit-Learning: Bool. Whether or not the rate limiter is enforcing the principal.

Anonymous clients

If your Spinnaker deployment allows anonymous API access, by default all requests will be bucketed into the same rate limit window. With a little trust, you can allow your users to assign themselves their own anonymous rate limit bucket by sending a X-RateLimit-App header, where the value is the name of their application.

Dynamic configuration

Re-deploying gate to respond to real-world events is non-ideal. Redis has the capability to override any configuration defined by the static configuration. First, a list of key patterns:

rateLimit* - All rate limiter-related keys.
rateLimit:* - Principal rate limiter.
rateLimit:learning - Global learning flag.
rateLimit:enforcing - A list of principals being enforced.
rateLimit:ignoring - A list of principals in learning mode.
rateLimit:capacity:* - Per-principal capacity.
rateLimit:rateSeconds:* - Per-principal rateSeconds.

Examples

$ redis-cli

# Set a new capacity
127.0.0.1:6379> set rateLimit:capacity:chaosmonkey@example.com 2000

# Add the chaosmonkey@example.com principal to the enforcing list
127.0.0.1:6379> sadd rateLimit:enforcing chaosmonkey@example.com
(integer) 1

Reference

Code backing the rate limiter can be found in the gate repository .

Docs: Caching: Account Name Based Sharding

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Account name based sharding is applicable to the caching function of a clouddriver. In case of HA mode, it is applicable to clouddriver-caching pods only. This feature is added at caching scheduler(both redis & sql backed) level which means it’s applicable to any cloud provider.

This is a configurable feature. When configured, all the caching agents of an account are run by a single caching pod. The logic of identifying the account whose caching agents are to be run in a particular pod is dependent on the name of the account(hashcode) and the number of available caching pods. When the pods are scaled up or down, the agents will resettle to the same or a different pod based on the sharding logic.

Configuration

This feature can be enabled by setting cache-sharding.enabled property to true which by default is false. The below configuration needs to be added to appropriate configuration file (ex: clouddriver-local.yaml):

cache-sharding:
  enabled: true

Additional Details

The implementation details are available at the following PR links.

feat(cats/sql): account name based sharding logic

feat(cats/redis): account name based sharding logic

refactor(cats/sql): made sharding configuration consistent

Docs: Orca: Quality of Service

Mon, 01 Jan 0001 00:00:00 +0000

EXPERIMENTAL: This feature is still in an early adoption / experimental phase. While you can use it today (Orca v6.71.0), Netflix is currently running this in learning mode / judiciously enabling in response to on-call events.

Spinnaker ships with an optional Quality of Service (QoS) module that can be used to manage the amount of active executions running at any given time. By default, this QoS module is disabled, but can be enabled and tuned with a handful of knobs and different strategies. Before we dive into the configuration settings, we’ll go over how QoS works.

How QoS Works

The QoS system is self-contained within Orca’s orca-qos module and operates only on newly created executions (both pipeline and orchestrations). That is to say, the QoS system will not impact running executions: Once an execution has started, it is completely outside of the domain of the QoS module. The rest of this section assumes that QoS is enabled.

When an execution is submitted to Orca (either manually via the API or UI, or through an automated trigger), Orca will first emit a synchronous BeforeExecutionPersist event which the QoS ExecutionBufferActuator is listening on. The behavior of the ExecutionBufferActuator depends firstly on the result of a BufferStateSupplier . The BufferStateSupplier can perform whatever heuristics necessary to determine whether or not any new execution should go through the QoS process. If the BufferStateSupplier returns false, no other QoS actions occur and the execution is started as normal.

In the event BufferStateSupplier returns true, the execution is passed through a chain of ordered BufferPolicy functions. These BufferPolicy functions return a result defining whether or not to BUFFER or ENQUEUE the execution. All BufferPolicy functions must return ENQUEUE, otherwise the execution will be assigned a status of BUFFERED, delaying the initialization of the execution. When an execution is BUFFERED, it will effectively stay in a waiting state until it is unbuffered, which we’ll go over later.

BufferPolicy functions are pluggable and can contain arbitrary logic. For example, one BufferPolicy that is always enabled is EnqueueDeckOrchestrationBufferPolicy , which will always ENQUEUE an execution it is an Orchestration and from the UI. This specific policy forces the ENQUEUE status, even if other policies call for the execution to be buffered; this is done through a force flag that policies can return. An example of other pluggable behavior is determining buffering action based on criticality of the execution: At Netflix we have a custom concept of application criticality, so we can buffer low criticality executions to allow capacity for higher criticality executions.

Once an execution is put into a BUFFERED state, it will remain in that state until the ExecutionPromoter decides to change it to NOT_STARTED and enqueue it for processing. Similar to the ExecutionBufferActuator, the ExecutionPromoter uses an ordered chain of PromotionPolicy functions to determine what buffered executions to promote. Every promotion cycle, the list of buffered executions (called candidates) are passed through the policies, each reducing the list of candidates to a final list of executions that will be promoted. Again, these policies can have arbitrary logic, but by default a naive promotion policy is included that will promote the N oldest executions. This promotion process happens (by default) on a 5-second interval on every Orca instance.

With both BufferPolicy and PromotionPolicy, the results of each function returns a result with a human readable “reason”, which is logged out for each execution that is evaluated so it is easy to trace.

sequenceDiagram participant ExecutionPersister participant ExecutionBufferActuator participant BufferPolicy participant ExecutionPromoter participant PromotionPolicy participant ExecutionLauncher ExecutionPersister->>ExecutionBufferActuator: BeforeExecutionPersistEvent ExecutionBufferActuator->>BufferPolicy: Execution loop Buffer Chain BufferPolicy->BufferPolicy: Evaluate if BUFFERED or ENQUEUED end alt ENQUEUED BufferPolicy->>ExecutionLauncher: Start Execution else BUFFERED Note over BufferPolicy: Set BUFFERED status end note right of ExecutionPromoter: Every n seconds ExecutionPromoter->>PromotionPolicy: All BUFFERED executions "candidates" loop PromoteCycle PromotionPolicy->PromotionPolicy: Reduce candidates end PromotionPolicy->>ExecutionPromoter: Final promotion candidates loop Promote Executions note over ExecutionPromoter: For each promoted execution ExecutionPromoter->>ExecutionPersister: Update Execution status to NOT_STARTED ExecutionPromoter->>ExecutionLauncher: Start Execution end

Note: This is the first implementation of the QoS system, we plan to iterate on this concept and make it more advanced over time. You can read the original proposal to get an idea of a potential roadmap.

Configuration

These configurations are not guaranteed to be fully inclusive of all knobs. A definitive list is available via the codebase.

qos.enabled: Boolean (default false). Global flag controlling whether or not the system is enabled. This flag will not disable the ExecutionPromoter.
qos.learningMode.enabled: Boolean (default true). If enabled, executions will always be ENQUEUED, but log messages & metrics will be emitted saying what the system would have done. This flag has no effect on ExecutionPromoter.
pollers.qos.promoteIntervalMs: Integer (default 5000). The time (in milliseconds) that the promotion process will be run.

BufferPolicy: Naive

The NaiveBufferPolicy will always buffer executions when enabled.

qos.bufferPolicy.naive.enabled: Boolean (default true).

BufferStateSupplier: ActiveExecutions

The ActiveExecutionsBufferStateSupplier will enable/disable the buffering state based on the number of active executions in the system.

qos.bufferingState.supplier must be set to activeExecutions.
qos.bufferingState.activeExecutions.threshold: Integer (default 100). The high threshold of active executions before QoS will start actuating on executions.
pollers.qos.updateStateIntervalMs: Integer (default 5000). The time (in milliseconds) that the function will update its internal record for how many executions are running in the system.

BufferStateSupplier: KillSwitch

The KillSwitchBufferStateSupplier will enable/disable the buffering state based on configuration only. This is handy if you’re evaluating the fundamentals of the QoS system, or you want a break-the-glass operator knob to control QoS.

qos.bufferingState.supplier must be set to killSwitch.
qos.bufferingState.killSwitch.enabled: Boolean (default false). If true, QoS will be enabled.

PromotionPolicy: Naive

The NaivePromotionPolicy will promote N executions every promotion cycle.

qos.promotionPolicy.naive.enabled: Boolean (default true). Whether or not this policy is enabled.
qos.promotionPolicy.naive.size: Integer (default 1). The max number of executions to promote.

Monitoring

qos.executionsBuffered: Counter. The number of executions that have been buffered.
qos.executionsEnqueued: Counter. The number of executions that have been enqueued (e.g. passed through the system and were judged not to be buffered).
qos.actuator.elapsedTime: Timer. The amount of time that is spent passing an execution through all enabled BufferPolicys.
qos.promoter.elapsedTime: Timer. The amount of time that is spent passing an execution through all enabled PromotionPolicys. Since the promoter is run on a static interval, this should usually be a relatively high, yet constant, number.
qos.promoter.executionsPromoted: Counter. The number of executions that have been promoted.

Additional Notes

The QoS system is currently shared-nothing state. Each Orca instance will maintain its own state (aside from configuration) about whether or not it should be buffering executions, or when it should be running pollers.

Docs: Orca: Zombie Executions

Mon, 01 Jan 0001 00:00:00 +0000

Aliases: orphaned execution

A zombie Execution is one that has a status in the database of RUNNING but there are no messages in Orca’s work queue or unacked set—the pipeline or task is not doing anything.

Diagnosis

Logs will be emitted regularly for Executions that are currently running in Orca via the QueueProcessor class, which will look similar to the following example. If no logs have been emitted for over 10 minutes for a RUNNING Execution, it is very likely a zombie.

Received message RunTask(executionType=pipeline, executionId=01CT1ST3MBJ9ECPH5JM5HVJARE, application=myapplication, stageId=01CT1ST4P79Y3MPW6FC4H38N3A, taskId=8, taskType=class com.netflix.spinnaker.orca.clouddriver.tasks.instance.WaitForUpInstancesTask)

Metrics & Alerting

Orca can be configured to detect and emit metrics for zombie Executions. This setting is expensive with the RedisExecutionRepository and is disabled by default. You can enable this detection by setting queue.zombieCheck.enabled: true in your Orca configuration.

When enabled, any discovered zombies will be logged out, as well as emitted via a metric:

Found zombie executionType=pipeline application=myapplication executionName=myexample executionId=01CS076X85RX6MWBTQ0VGBF8VX

If you’ve enabled the zombie check, set an alert on the metric queue.zombies, triggering whenever there is a count greater than 0.

Remediation

Rehydrate the Queue

If the Execution is a zombie, there are no messages on the work queue for that Execution. You can attempt to re-hydrate the queue — reissue messages onto the work queue based on the last stored state — using an admin API in Orca , which must be called directly as it is not exposed through Gate. This command can take either a single execution or operate on all executions within a time range. This command will dry-run by default. To actually rehydrate the queue, pass the query parameter dryRun=false.

$ curl -XPOST \
  https://localhost:8083/admin/queue/hydrate?executionId=01CS076X85RX6MWBTQ0VGBF8VX&dryRun=false

This command is best effort and may not be able to rehydrate the Execution, especially if the Execution was zombied while running a non-retryable task.

An example response from the endpoint:

{
  "dryRun": false,
  "executions": {
    "01CS076X85RX6MWBTQ0VGBF8VX": {
      "startTime": 1538679600852,
      "actions": [
        {
          "description": "Task is running and is retryable",
          "message": {
            "kind": "runTask",
            "executionType": "PIPELINE",
            "executionId": "01CS076X85RX6MWBTQ0VGBF8VX",
            "application": "myapplication",
            "stageId": "01CS076X8501MNAD2ZTJ4ST2TM",
            "taskId": "1",
            "taskType": "com.netflix.spinnaker.orca.echo.pipeline.ManualJudgmentStage$WaitForManualJudgmentTask",
            "attributes": [],
            "ackTimeoutMs": 600000
          },
          "context": {
            "stageId": "01CS076X8501MNAD2ZTJ4ST2TM",
            "stageType": "manualJudgment",
            "stageStartTime": 1538682406227,
            "taskId": "1",
            "taskType": "waitForJudgment",
            "taskStartTime": 1538682406242
          }
        },
        {
          "description": "Task is running but is not retryable",
          "context": {
            "stageId": "01CS076X85ECXHF3FRWZBTQ359",
            "stageType": "createProperty",
            "stageStartTime": 1538681485559,
            "taskId": "3",
            "taskType": "monitorProperties",
            "taskStartTime": 1538681546116
          }
        }
      ],
      "canApply": false
    }
  }
}

For each Execution, a final action summary is provided canApply. If any part of an Execution cannot be re-hydrated, the entire Execution will be skipped.

Cancel the Execution

If the Execution cannot be rehydrated, it will need to be canceled. You can cancel the Execution via the UI or force cancellation via an Orca admin API:

PUT /admin/forceCancelExecution?executionId=01CS076X85RX6MWBTQ0VGBF8VX&executionType=PIPELINE

Known Causes

Zombie Executions can occur due to a loss of the Redis instance backing the Orca queue, or in prolonged unreliable networks. If you’re using the RedisExecutionRepository, it’s likely that when you lose the Redis backing the queue, you’re also losing all running Executions. However, while using the SQL backend for Orca , losing the Redis means you’re only losing the in-flight work queue state, not the state of the pipelines. In such a scenario, once Redis has been restored it will not have any messages to process and existing RUNNING Executions will sit unprocessed.

Docs: Sharding Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Intro

This document shows you how to shard traffic to different Spinnaker services based upon configured criteria. The general pattern is to define a selector class in your configuration. Endpoints will then be selected based upon the criteria specified in the selectors.

As an example, Netflix creates read-only shards for Clouddriver to better manage requests.

Selectors exist at these levels:

Application
Execution type (i.e, Pipeline vs Orchestration)
Origin
Authenticated User
Deployment account
Cloud Provider

You will need to modify your Spinnaker deployment pipelines to ensure the infrastructure for each shard is correctly created, otherwise traffic could be sent to non-existent shards.

If no selector is specified, the default endpoint will be used.

Additionally, there is a special dynamicEndpoints configuration in gate.yml to send all requests from Deck to that particular shard.

Sharding Orca Requests

In gate.yml

services:
  orca:
    shards:
      baseUrls:
        - baseUrl: https://orca.example.com
        - baseUrl: https://orca-shard1.example.com
          priority: 10
          config:
            selectorClass: com.netflix.spinnaker.kork.web.selector.ByApplicationServiceSelector
            applicationPattern: xxxxyyyapp |demo.*xxxxyyyy

Clouddriver Read-only Shards

gate.yml

services:
  clouddriver:
    baseUrl: https://clouddriver-readonly.example.com
    config:
      dynamicEndpoints:
        deck: https://clouddriver-readonly-deck.example.com

orca.yml

clouddriver:
  readonly:
    baseUrls:
    - baseUrl: https://clouddriver-readonly-orca-1.example.com
      priority: 10
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByExecutionTypeServiceSelector
        executionTypes:
          - orchestration
    - baseUrl: https://clouddriver-readonly-orca-2.example.com
      priority: 20
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByApplicationServiceSelector
        applicationPattern: app1|.*app2.*
    - baseUrl: https://clouddriver-readonly-orca-3.example.com
      priority: 30
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByOriginServiceSelector
        origin: deck
        executionTypes:
          - orchestration
    - baseUrl: https://clouddriver-readonly-orca-4.example.com
      priority: 50
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByAuthenticatedUserServiceSelector
        users:
          - horseman.*
          - bojack.*
    - baseUrl: https://clouddriver-readonly-orca-5.example.com

Clouddriver Write-only Shards

orca.yml

clouddriver:
  writeonly:
    baseUrls:
    - baseUrl: https://clouddriver-write-kubernetes.example.com
      priority: 10
      config:
        selectorClass: com.netflix.spinnaker.kork.web.selector.ByCloudProviderServiceSelector
        cloudProviders: 
          - kubernetes
    - baseUrl: https://clouddriver-write-cloudfoundry.example.com
      priority: 20
      config:
        selectorClass: com.netflix.spinnaker.kork.web.selector.ByCloudProviderServiceSelector
        cloudProviders: 
          - cloudfoundry
    - baseUrl: https://clouddriver-write-heavy-account.example.com
      priority: 30
      config:
        selectorClass: com.netflix.spinnaker.kork.web.selector.ByAccountServiceSelector
        accountPattern: heavyUsageAccount.*|.*app2.*
    - baseUrl: https://clouddriver-write-heavy-users.example.com
      priority: 40
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByAuthenticatedUserServiceSelector
        users:
          - horseman.*
          - bojack.*
    - baseUrl: https://clouddriver-write.example.com