Spinnaker – Operator Guides

Docs: Custom CAs for Webhooks

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Webhook stages enable Spinnaker to make HTTP(S) calls to external web services. If the configured webhook URL has the https:// scheme, Spinnaker will use TLS to communicate with the external server. Spinnaker will attempt to validate the certificate presented by the server by building a chain of trust back to a trusted certification authority (CA) and will refuse to connect if the certificate cannot be validated.

By default Spinnaker uses the trust store provided by the JVM as its source of trusted CAs. The default behavior is sufficient for webhooks to public-facing servers where it is possible to build a chain of trust back to a root CA. Internal servers, however, may have certificates issued by a company-specific CA that is not trusted by a root CA. Webhooks to these servers over https:// will fail using the default configuration.

In order to support this latter use case, Spinnaker allows users to supply additional CAs to trust in addition to the default ones. These additional CAs will be used when negotiating connections for outbound webhooks (including preconfigured webhooks) but will not be used for any other connection initiated by Spinnaker. There is no way to specify additional CAs on a per-webhook basis; the additional CAs will apply to all webhooks.

Create a trust store

Create a trust store in Java KeyStore (JKS) format via:

keytool -import -file <path-to-ca-certificate> -alias <name-of-first-ca> -keystore <name-for-keystore>.jks

where <path-to-ca-certificate> is the path to the certificate for the CA you’d like to trust in PEM format, <name-of-first-ca> is an arbitrary alias for that CA, and <name-for-keystore> is the name of a keystore that will be created.

You will be prompted to create a password for the new key store, which you’ll need to supply to Spinnaker in the next step.

After creating the key store with the above command, you can add additional CAs to the keystore by running the same command but supplying a different CA certificate and alias. You’ll be prompted for the keystore password before the new CA can be added. As this trust store will augment the default trust store, you don’t need to add all of the root CAs to this custom trust store; only CAs that are not in the default trust store need to be added.

Configure Spinnaker to use the trust store

Aadd the following can be added to orca-local.yml:

webhook:
  trust:
    enabled: true
    trustStore: <path to trust store in jks format>
    trustStorePassword: <password for trustStore>

Note: Using a secrets manager for these entries is recommended.

Docs: Custom Job Stages

Mon, 01 Jan 0001 00:00:00 +0000

For users of Kubernetes or Titus, Custom Job Stages offer a native solution for enhancing Spinnaker’s functionality with the power of containers. Custom Job Stages enable operators to extend Spinnaker without having to maintain a custom build of Orca.

Note: Custom Job stages are only supported for the Kubernetes and Titus providers.

How it works

At its core, Custom Job Stages use the Run Job stage to start a Task (Titus) or Job (Kubernetes) to perform some type of work. These jobs are defined as configuration for Orca and registered when the application starts. When used in a pipeline, Orca takes this configuration, generates a Run Job stage and then executes it. To support a more native experience, we provide configuration options for things like stage name, description and parameters so that this stage feels less like a Run Job stage and more like an out of the box stage.

When should I use Custom Job Stages?

Custom Job Stages are perfect for when you need to encapsulate a common task so that all users of Spinnaker can benefit and standardize. For example, if you see a large number of users running database migrations in the same way, but manually defining jobs (via Run Job), simply define a Custom Job Stage which captures all the complexity of the job and exposes a subset of customization options.

Some example use cases for Custom Job Stages include:

Building Docker images with Kaniko
Running integration tests
Interacting with external systems
Performing database migrations

Creating a Custom Job Stage

Custom Job stages are configured as part of Orca’s configuration. To enable them, simply add configurations via the job.preconfigured.{cloudProvider} block in orca-local.yml where cloudProvider is the name of the cloud provider which will be used to run the job. For example, if you are using Kubernetes, the value will be kubernetes. The following properties are supported for all providers.

enabled - Determines if stage will be available in the UI.
label - Human readable stage name.
description - Human readable description of what the stage does.
type - Unique key used to determine stage type.
cloudProvider - Cloud provider to use for this stage. Valid values are titus or kubernetes.
credentials - Account name used when executing the job. Must be valid account for cloudProvider.
waitForCompletion - If true, the stage will not be marked complete until the job completes.
parameters - List of parameters presented to the end user.

Custom Job Stages - Kubernetes

The following properties are supported for the kubernetes cloud provider.

account - Account name used when executing the job. Must be valid account for cloudProvider.
application - Name of the application to associate the job with.
manifest - YAML definition for the Kubernetes Job that will be run when the stage is executed.
propertyFile - The container name to capture output data from. Captured output will be available in the pipeline context for use in downstream stages.

Note: If you’re using the slash (/) character in any manifest annotations, you’ll need to use this special syntax to prevent the slash from being dropped when the application reads its configuration. For example, if your annotation key is iam.amazonaws.com/role you’ll need to define it like so: [iam.amazonaws.com/role].

Custom Job Stages - Titus

cluster - Definition of the cluster to be created when the stage is executed.
region - Region in which to run the underlying job.
propertyFile - File to download once the job completes.

Custom Job Stage parameters

Using parameters with the Custom Job Stage enables your users to customize specific fields of the job configuration at runtime. Before the stage is started, user supplied values will replace the values in the stage’s configuration. Parameters support the following properties

name - Name of parameter.
label - Human readable parameter name (shown in the UI).
defaultValue - default value, if not overridden by user.
description - Description of what parameter is used for.
order - Display order of the parameter in the UI.

Example Custom Job Stage

job:
  preconfigured:
    kubernetes:
      - label: Echo
        type: echoPhrase
        description: Echo a configured phrase
        cloudProvider: kubernetes
        account: kubernetes
        credentials: kubernetes
        waitForCompletion: true
        application: k8s2
        parameters:
          - name: PHRASE
            label: Phrase to say
            description: Phrase to be echoed.
            mapping: manifest.spec.template.spec.containers[0].env[0].value
            defaultValue: "Hello world!"
        manifest:
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: say-hello
            namespace: spinnaker
          spec:
            backoffLimit: 0
            template:
              spec:
                restartPolicy: Never
                containers: 
                  - name: hello
                    image: alpine:latest
                    env:
                      - name: PHRASE
                        value: placeholder
                    command: ["echo", $(PHRASE)]

Using a Custom Job Stage

Custom Job Stages will appear as native stages within the Spinnaker UI. Users can add these stages to their pipelines as they would any other stage.

If the stage defines any variables they will be rendered and presented to the user. At this point, the user is free to override the values as they need.

Custom Job stages also take on a native look and feel while executing. The below screen shots show an example of the Echo stage running. You’ll notice that logs are presented to the user for ease of debugging.

Docs: Custom Webhook Stages

Mon, 01 Jan 0001 00:00:00 +0000

Custom webhook stages provide a simple, yet powerful, way of adding custom stages to Spinnaker. These stages are typically used to make quick API calls to an external system as part of a pipeline. Instead of extending the various components through code, users can simply add configuration to Orca for these stages. They appear in the UI as if they were a native stage.

Creating a Custom Webhook Stage

To set up a custom webhook stage, you’ll need to modify the orca-local.yml configuration file. The webhook.preconfigured property allows you to define multiple webhook stages. Here’s an example configuration:

webhook:
  preconfigured:
    - label: GitHub - Update Commit Status
      type: githubStatus
      enabled: true
      description: Update a GitHub Commit Status
      method: POST
      url: https://api.github.com/repos/your-repo/statuses/${parameterValues['gitCommit']}
      customHeaders:
        Authorization:
          - token YOUR_API_TOKEN
      payload: |-
        {
          "state": "${parameterValues['status']}",
          "target_url": "${parameterValues['targetUrl']}",
          "context": "${parameterValues['context']}"
        }        
      parameters:
        - label: Git Commit
          name: gitCommit
          description: The Git commit SHA
          type: string
        - label: Status
          name: status
          description: The state of the status (e.g., success, failure)
          type: string
        - label: Target URL
          name: targetUrl
          description: The URL associated with this status
          type: string
        - label: Context
          name: context
          description: A string label to differentiate this status from others
          type: string

In this configuration:

label: The name displayed in the UI for the stage.
type: A unique identifier for the stage.
enabled: Determines if the stage is available in the UI.
description: A brief explanation of the stage’s purpose.
method: The HTTP method used for the webhook (e.g., GET, POST).
url: The endpoint for the webhook, which can include parameters.
customHeaders: Any headers required for the HTTP request, such as API tokens.
payload: The JSON payload sent with the request.
parameters: Defines user-input fields in the UI, which can be referenced in the url and payload using SpEL expressions.

Note: Once a property is set within a custom webhook stage, users cannot override that setting via the UI.

Configuring Parameters for Custom Webhook Stages

Parameters allow users to input values when configuring the stage in a pipeline. These values can be dynamically accessed using SpEL (Spring Expression Language) expressions. For example, to set up a parameter for a Git commit SHA:

parameters:
  - label: Git Commit
    name: gitCommit
    description: The Git commit SHA
    defaultValue: ''
    type: string

Currently, only the string type is supported for parameters.

Using a Custom Webhook Stage

Once configured, your custom webhook stage will appear in the list of available stages within the Pipeline editor. You can add it to your pipeline like any other stage. If you’ve specified any parameters, they will be rendered and editable as part of the stage configuration. You can also use SpEL to set these values dynamically during pipeline execution.

Useful Custom Webhook Stage Examples

Update GitHub Commit Status

This stage updates a commit status in GitHub as part of a pipeline execution.

label: GitHub - Update Commit Status
type: githubStatus
enabled: true
description: Update a GitHub Commit Status
method: POST
customHeaders:
  Authorization:
    - token YOUR_API_TOKEN
url: https://api.github.com/repos/your-repo/statuses/${parameterValues['gitCommit']}
payload: |-
  {
    "state": "${parameterValues['status']}",
    "target_url": "${parameterValues['targetUrl']}",
    "context": "${parameterValues['context']}"
  }  
parameters:
  - label: Git Commit
    name: gitCommit
    description: The Git commit SHA
    type: string
  - label: Status
    name: status
    type: string
  - label: Target URL
    name: targetUrl
    type: string
  - label: Context
    name: context
    type: string

Create a New Relic Deployment

This stage sends deployment details to New Relic for tracking.

label: New Relic - Create Deployment
type: newRelicDeployment
enabled: true
description: Create a Deployment in New Relic
method: POST
customHeaders:
  X-Api-Key:
    - YOUR_API_KEY
  Content-Type:
    - application/json
url: https://api.newrelic.com/v2/applications/${parameterValues['appId']}/deployments.json
payload: |-
  {
    "deployment": {
      "revision": "${parameterValues['revision']}",
      "user": "${parameterValues['user']}",
      "description": "${parameterValues['description']}"
    }
  }  
parameters:
  - label: Application ID
    name: appId
    description: New Relic Application ID
    type: string
  - label: Revision
    name: revision
    type: string
  - label: User
    name: user
    type: string
  - label: Description
    name: description
    type: string

Testing Webhook Stages

To test your webhook stages before adding them to your pipeline, you can use services like:

Beeceptor – Set up a mock API endpoint to inspect webhook requests.
RequestBin – Capture and analyze HTTP requests to debug your webhooks.

These tools help ensure your webhook stages are correctly configured before deploying them in Spinnaker.

Docs: Echo: Cassandra to In-Memory

Mon, 01 Jan 0001 00:00:00 +0000

Echo’s scheduler can be run completely in-memory. On startup or redeploy, echo will check cron schedules to see if it needs to retroactively execute any missed triggers. This migration only requires configuration changes.

1. Disable Cassandra in echo.yml

spinnaker:
  cassandra:
    enabled: false

2. Enable in-memory backend in echo.yml

spinnaker:
  inMemory:
    enabled: true

3. Enable the scheduler compensation job in echo.yml

scheduler:
  compensationJob:
    enabled: true
    windowMs: 1800000 # optional

The windowMs property dictates how far in the past echo will look to find missed schedules. By default this is 30 minutes.

4. Deploy new Echo

Docs: Front50: Cassandra to Object Store (S3, Azure, or GCS)

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra is no longer an actively maintained or supported persistence store.

A migration to S3, GCS, or AZS is recommended.

1. Create a bucket and folder

Make up a bucket name (or account name if using Azure) that is consistent with the naming policies for the underlying storage service. For purposes of this document, we’ll pick ${USER}-spinnaker since many storage services (including Amazon S3, Google GCS, and Azure AZS) require globally unique names.

Make up a folder name within the bucket. The default is “front50”. Spinnaker will store all the objects within this folder. The folder name will be used for configuration and managed by Spinnaker. You do not need to physically create the folder.

Note that currently only Amazon Simple Storage Service (S3), Google Cloud Storage (GCS), or Azure Storage (AZS) are supported, and they are mutually exclusive. Pick only one. This is independent of where you are actually running Spinnaker.

A. Create the bucket in S3

If you wish to use S3 as the storage service, create the bucket and underlying root folder.

See Amazon’s Documentation for more information.

To enable versioning on an existing bucket, follow these steps.

B. Create the bucket in GCS

If you wish to use GCS as the storage service, Spinnaker can automatically create the bucket (with versioning) for you if it has the right OAuth scopes (Storage Admin). To create it yourself you are best off using the gsutil tool . Using gsutil, also turn on versioning within the bucket.

gsutil mb ${USER}-spinnaker
gsutil versioning set on ${USER}-spinnaker

C. Create the storage account in AZS

If you wish to use Azure Storage, Spinnaker can automatically create the container and root folder (with versioning). Follow instructions here to create a storage account and make sure to note the storage account name and a key.

2. Disable Cassandra in spinnaker-local.yml

services:
  front50:
    cassandra:
      enabled: false

3. Enable the object store in spinnaker-local.yml

A. Enable S3

If you wish to use S3 as the storage service, set the following:

services:
  front50:
    storage_bucket: YOUR_S3_BUCKET_NAME (From Step #1)
    bucket_root: YOUR_S3_FOLDER_NAME (From Step #1)
    s3:
      enabled: true

B. Enable GCS

If you wish to use GCS as the storage service, set the following YAML properties. The default project name is the ${providers.google.primaryCredentials.project} from spinnaker-local.yml. It is needed only if the bucket does not exist and Spinnaker is going to create the bucket for you. The jsonPath is used to convey the credentials to use in order to access GCS. The default is ${providers.google.primaryCredentials.jsonPath} from spinnaker-local.yml however if you created the bucket in a different project (e.g. in the project that is running Spinnaker as opposed to the project that Spinnaker may be managing) then you may need to supply different credentials.

For more information about credentials and obtaining a JSON credentials file, see the discussion on Service Accounts

services:
  front50:
    storage_bucket: YOUR_GCS_BUCKET_NAME (From Step #1)
    bucket_root: YOUR_GCS_FOLDER_NAME (From Step #1)
    gcs:
      enabled: true
      project: SEE NOTE
      jsonPath: SEE NOTE

C. Enable AZS

If you wish to use Azure Storage, set the following:

services:
  front50:
    azs:
      enabled: true
      storageAccountName: YOUR_STORAGE_ACCOUNT_NAME (From Step #1)
      storageAccountKey: YOUR_STORAGE_ACCOUNT_KEY (From Step #1)
      storageContainerName: front50

4. Export existing applications, pipelines, strategies, notifications and projects

Replace “FRONT50_HOSTNAME” and “FRONT50_PORT” and run the following (“localhost” and “8080” are the defaults):

#!/bin/sh

rm applications.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/v2/applications | json_pp > applications.json

rm pipelines.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/pipelines | json_pp > pipelines.json

rm strategies.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/strategies | json_pp > strategies.json

rm notifications.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/notifications | json_pp > notifications.json

rm projects.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/v2/projects | json_pp > projects.json

5. Deploy new Front50

Run sudo restart front50

6. Import Applications, Pipelines, Strategies, Notifications and Projects

Replace “FRONT50_HOSTNAME” and “FRONT50_PORT” and run the following (“localhost” and “8080” are the defaults):

#!/bin/sh

curl -X POST -H "Content-type: application/json" --data-binary @"notifications.json" http://FRONT50_HOSTNAME:FRONT50_PORT/notifications/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"strategies.json" http://FRONT50_HOSTNAME:FRONT50_PORT/strategies/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"pipelines.json" http://FRONT50_HOSTNAME:FRONT50_PORT/pipelines/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"applications.json" http://FRONT50_HOSTNAME:FRONT50_PORT/v2/applications/batch/applications
curl -X POST -H "Content-type: application/json" --data-binary @"projects.json" http://FRONT50_HOSTNAME:FRONT50_PORT/v2/projects/batchUpdate

Docs: Front50: Cassandra to Redis

Mon, 01 Jan 0001 00:00:00 +0000

1. Install Redis

2. Disable Cassandra in front50.yml

cassandra:
  enabled: false

3. Enable Redis in front50.yml

spinnaker:
  redis:
    enabled: true

4. Export existing applications, pipelines, strategies, notifications and projects

#!/bin/sh

rm applications.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/global/applications | json_pp > applications.json

rm pipelines.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/pipelines | json_pp > pipelines.json

rm strategies.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/strategies | json_pp > strategies.json

rm notifications.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/notifications | json_pp > notifications.json

rm projects.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/v2/projects | json_pp | jq '._embedded.projects' > projects.json

5. Deploy new Front50

6. Import applications, pipelines, strategies, notifications and projects

#!/bin/sh

curl -X POST -H "Content-type: application/json" --data-binary @"notifications.json" http://FRONT50_HOSTNAME:FRONT50_PORT/notifications/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"strategies.json" http://FRONT50_HOSTNAME:FRONT50_PORT/strategies/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"pipelines.json" http://FRONT50_HOSTNAME:FRONT50_PORT/pipelines/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"applications.json" http://FRONT50_HOSTNAME:FRONT50_PORT/global/applications/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"projects.json" http://FRONT50_HOSTNAME:FRONT50_PORT/v2/projects/batchUpdate

Docs: Hiding Stages

Mon, 01 Jan 0001 00:00:00 +0000

Stages that are not provider-specific will be available by default. To hide specific stages from end-users, set the hiddenStages property in Deck’s settings-local.js to a list of the keys of stages you wish to hide. For example, to hide the Gremlin and Travis stages, include the following in settings-local.js:

 window.spinnakerSettings.hiddenStages = ['gremlin', 'travis'];

Docs: Orca: Redis to SQL Migration

Mon, 01 Jan 0001 00:00:00 +0000

If you are not migrating an existing Orca deployment, refer to Orca SQL Setup instead.

Migrate from Redis to SQL

Migrating without downtime from Redis to SQL is a three-step process:

When DualExecutionRepository is running, writes will be routed to either Redis or SQL. Executions will only be migrated to SQL once they’ve completed (either successfully or terminally): This keeps the migration story simple. As such, the migration agents will need to run for awhile. At Netflix, we ran the migration cluster for two weeks, as we had long pipeline executions due to canaries. You may only need to run the migration cluster for an hour.

NOTE: Deploying the migrators as a separate cluster is optional, however the migration process is memory hungry, so you may need to devote more resources to the Orca process.

Enable DualExecutionRepository

Building atop the baseline configuration above, add the following to orca.yml:

executionRepository:
  dual:
    enabled: true
    primaryName: sqlExecutionRepository
    previousName: redisExecutionRepository
  sql:
    enabled: true
  redis:
    enabled: true

Note that both repositories are enabled. Orca will fail to start up if the DualExecutionRepository is misconfigured.

Deploy a Migration Cluster

At Netflix, we deployed orca-main, the cluster that serves our production traffic, as well as orca-main-sqlmigration, which does not receive API traffic nor process the work queue. Its sole purpose is to shovel bits from Redis to SQL.

To perform a deploy a migration cluster, add the following configuration to orca.yml:

---
spring:
  profiles: sqlmigration

pollers:
  orchestrationMigrator:
    enabled: true
    intervalMs: 1800000
  pipelineMigrator:
    enabled: true
    intervalMs: 1800000

queue:
  redis:
    enabled: false

keiko:
  queue:
    enabled: false

You will need to launch this migration cluster with -Dspring.profiles.active=sqlmigration. Spring Profiles allow you to configure a service to startup with different configurations.

Disable DualExecutionRepository

Once all executions have been migrated, you can deploy Orca once again without DualExecutionRepository and delete the migration cluster.

SQL-specific Metrics

Orca emits a few handfuls of metrics that are specific to SQL.

The SqlHealthcheckQueueActivator class, which will disable work queue processing if SQL connectivity goes unhealthy, emits one metric: sql.queueActivator.invocations with a tag of status (disabled or enabled).

The ExecutionRepository will emit a bunch of invocation and timing metrics with the following patterns:

sql.executionRepository.$method.timing
sql.executionRepository.$method.invocations

If you are using the default HikariCP connection pool:

sql.pool.$poolName.connectionAcquiredTiming
sql.pool.$poolName.connectionUsageTiming
sql.pool.$poolName.connectionTimeout
sql.pool.$poolName.idle
sql.pool.$poolName.active
sql.pool.$poolName.total
sql.pool.$poolName.blocked

If you are using the MariaDB driver extension:

sql.pool.$poolName.active
sql.pool.$poolName.idle
sql.pool.$poolName.total
sql.pool.$poolName.blocked