Spinnaker – Documentation

Docs: AWS EC2 Launch Templates Setup

Mon, 01 Jan 0001 00:00:00 +0000

Please note that you should only proceed with this if you have AWS EC2 configured as a cloud provider. These features require 1.24 (although some features were launched in previous releases leading up to 1.24).

AWS uses launch templates to specify instance configuration information. Launch templates are the successor of launch configurations. This means that any new instance configuration feature from AWS will only be supported by launch templates.

Spinnaker still supports launch configurations for backwards compatibility, but recommends enabling launch templates to access any new features that AWS adds.

Setup Steps

This section summarizes the steps required to set up launch templates if you are new to using AWS in Spinnaker or if you have already been using AWS as one of your cloud providers.

New to AWS

If you are new to Spinnaker or even just new to AWS in Spinnaker, we recommend immediately enabling launch template support for all applications.

Update your Clouddriver configuration file, usually clouddriver.yml, to enable launch templates for all applications.

  aws.features.launch-templates.enabled: true
  aws.features.launch-templates.all-applications.enabled: true

Read through the available launch template supported features to determine which features make sense for your users.

Update AWS settings in settings-local.js of Deck to enable launch templates and the features you identified. Ensure that enableLaunchTemplates is true.

window.spinnakerSettings.providers.aws.serverGroups = {
    enableLaunchTemplates: true,
    enableIPv6: true,
    enableIMDSv2: true,
    enableCpuCredits: true,
};

Current AWS User

If you already use AWS as a cloud provider in Spinnaker, we recommend migrating to launch templates. Since there may be pre-existing dependencies on launch configurations, we have created some rollout configurations you can utilize for testing and/or migration.

Update clouddriver.yml. This step can be repeated as needed throughout your rollout. This is an example config where launch templates is rolled out to two applications in production and all of the test account. It also excludes one application completely:
```
  aws.features.launch-templates.enabled: true
  aws.features.launch-templates.allowed-applications: "myapp:prod:us-east-1,anotherapp:prod:us-east-1"
  aws.features.launch-templates.allowed-accounts: "test"
  aws.features.launch-templates.excluded-applications: "dangerousapp"
```
Review the rollout configurations and determine which of these you can temporarily utilize for your rollout. If you do not need to rollout, stop here and follow the new AWS users steps instead.
Read through the available launch template supported features to determine which features make sense for your users.

Update AWS settings in settings-local.js of Deck to enable launch templates and the features you identified. Ensure that enableLaunchTemplates is true.

// enable launch templates for AWS
window.spinnakerSettings.providers.aws.serverGroups = {
    enableLaunchTemplates: true,
    enableIPv6: true,
    enableIMDSv2: true,
    enableCpuCredits: true,
};

When you are ready for a complete rollout, enable launch templates for all applications and clean up rollout config in clouddriver.yml.
```
aws.features.launch-templates.enabled: true
aws.features.launch-templates.all-applications.enabled: true
```

Rollout Configuration

If you already use AWS, then your applications may have some dependencies on launch configurations that prevent simple feature enabling. The configuration options below were created to aid with testing or a rollout period. Feel free to use whatever combination is best for you. If you would prefer to skip a rollout, use the configuration in New to AWS .

Config	Type	Description	Example
allowed-applications	String	A comma-separated list of one or more allowed applications scoped by account-region pairs ("app:account:region"). This helps with preliminary controlled testing on a handful of applications.	"testapp:prod:us-east-1"
allowed-accounts-regions	String	A comma-separated list of allowed account-region pairs. This is good for incrementally rolling out to regions within accounts.	"test:us-east-1"
allowed-accounts	String	A comma-separated list of allowed accounts. This is good for incrementally rolling out launch templates from test to production accounts.	"test"
excluded-accounts	String	A comma-separated list of accounts to exclude from rollout.	"prod"
excluded-applications	String	A comma-separated list of applications to exclude from rollout. This helps prevent any edge cases from delaying a wide rollout..	"myapp1,myapp2"
all-applications.enabled	Boolean	Allows launch templates on any application, except for those that have been excluded. This will override any of the allowed lists, and widely rollout launch templates.	true

Feature Configuration

Learn more about the feature set along with sample API requests here .

Docs: Canary Overview

Mon, 01 Jan 0001 00:00:00 +0000

Canary is a deployment process in which a change is partially rolled out, then evaluated against the current deployment (baseline) to ensure that the new deployment is operating at least as well as the old. This evaluation is done using key metrics that are chosen when the canary is configured.

Canaries are usually run against deployments containing changes to code, but they can also be used for operational changes, including changes to configuration.

The canary process is not a substitute for other forms of testing.

Prerequisites

Have metrics to evaluate

Your application might send performance metrics which are published and available by default. You can also install a monitoring agent to collect more comprehensive metrics, and you can instrument your code to generate further metrics for that agent. In any case, you need to have access to a set of metrics, using some telemetry provider, which Kayenta can then use to make the canary judgment.

Support is built in for Stackdriver , Datadog , Prometheus , Signalfx , and New Relic .

Set up your canary environment

Before you can configure canary analysis and create canary stages for your pipelines, your Spinnaker administrator needs to enable canary for your installation .

How to make Canary work in Spinnaker—the high-level process

In Spinnaker, create one or more canary configurations .

The configuration provides the set of metrics for use in all pipeline canary stages that reference it, plus default scoring thresholds and weights—defaults that can be overridden in a canary stage

You can configure each metric flexibly, to define its scope and whether it fails when it deviates upward or down. You can also group metrics logically and assign weights to groups to control their impact on the final score.

You can think of this configuration as a templated set of queries against your metric store.
In any deployment pipeline that will use canary, add one or more canary stages .

The canary stage includes information that scopes the templated query (canary config) to a specified set of resources and time boundaries.

Docs: Configuring the Script Stage

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

The Script stage lets you run an arbitrary shell, Python, or Groovy script on a Jenkins instance as a first class stage in Spinnaker. For example, you can use it to launch a test suite from a pipeline instead of doing it manually. Using the Script stage requires some additional setup after you add Jenkins to your Spinnaker deployment.

Prerequisites

To configure a Script stage, you need:

A running Jenkins instance at $JENKINS_HOST, with a Jenkins user profile set up with access to create jobs
A running Spinnaker instance, with access to configuration files, and in which you have set up Jenkins on Spinnaker

Configure Jenkins

Download the raw job xml config file with the command:

curl -X GET \
    -o "scriptJobConfig.xml" \
    "https://storage.googleapis.com/jenkins-script-stage-config/scriptJobConfig.xml"

Create the Jenkins job where your script will run. To create the job, you need the following information:
- $JENKINS_HOST: your running Jenkins instance with protocol (http/https - e.g. https://myjenkins.com or http://somejenkins:8080).
- $JOB_NAME: the name of the Jenkins job where your script runs.
- $USER: your Jenkins username.
- $USER_API_TOKEN: the API token for your user. You can find this in Jenkins in the Configure page for your user (e.g. $JENKINS_HOST/user/$USER/configure).
Then, run the command:
```
curl -s -XPOST "$JENKINS_HOST/createItem?name=$JOB_NAME" \
    -u $USER:$USER_API_TOKEN \
    --data-binary @scriptJobConfig.xml \
    -H "Content-Type:text/xml"
```
Navigate to Jenkins » the job you just created » Configure and do the following:
1. Add the GitHub repository containing your scripts.
2. Either create a Spinnaker node in which Jenkins will run all Script jobs, or de-select the Restrict where this project can be run checkbox.
At this point, you can manually run the script job in Jenkins (including manually adding necessary parameters) and see it succeed.
If your Jenkins master is named anything other than master in your Spinnaker configuration, you’ll need to add the following to orca-local.yml in order for Spinnaker to find it:
```
script:
  master: your-jenkins-master
  job: $JOB_NAME  # from step #3
```

You can now use the Script stage in your pipelines.

Docs: Enable Slack Support Channels

Mon, 01 Jan 0001 00:00:00 +0000

You can create an application and configure it with relevant data—application attributes.

Including a Slack support channel for an application is a configurable application attribute. This feature communicates with Slack’s conversations API to expose a list of available Slack channels in your workspace. The Slack support channel feature is supported by all providers.

Configuration

Gate

You need an API token associated with your Slack workspace for gate to receive a list of channels from Slack’s API. See the Slack documentation if you need to create a Slack token. Add this token in the config for gate’s encrypted secrets.

slack:
  token: YOUR_TOKEN_HERE

The base url for the Slack API is also needed in gate.yml. This does not need to be encrypted.

slack:
  baseUrl: https://slack.com/api

Deck

The UI components in deck are protected with a feature flag in the application settings, settings.netflix.ts. To activate this feature you need to enable this flag to true.

feature: {
  slack: true,
  ...otherFeatures
}

If your workspace’s base URL is anything other than https://slack.com , then you need to configure this as well. This base URL is used to construct the link for accessing a Slack support channel.

  slack: {
    baseUrl: 'https://my-cool-workspace.slack.com',
  }

Docs: Get Started Using Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

This article assumes you are new to Spinnaker. It contains basic information about what to do after Spinnaker is installed .

Before you get going…

Whether you’re an operator or admin installing or managing Spinnaker or an end user getting started using Spinnaker, here are some pointers to get you started.

Operators (managing Spinnaker)

If you want to install Spinnaker, see the instructions here .
If you want to find out more about tuning and maintaining Spinnaker, here are some things to check out:
- Advanced configuration
- Productionize Spinnaker

Users (deploying with Spinnaker)

If you want to try out Spinnaker, please read the installation guide
If Spinnaker is already installed in your organization, and you want to practice it using some guided tutorials, here are codelabs .
If you want a basic overview of how to use Spinnaker, the high-level process is described below.

Using Spinnaker: the high-level process

Create an application

Typically, you’ll have one application per microservice.

Note: Your application configuration affects what you can do with Spinnaker—you can enable or disable some Spinnaker features.
Define the infrastructure the service will run on

You define infrastructure for each application. Your pipelines deploy services to the server groups you define.

Note: This step is not a prerequisite for creating pipelines. In fact, you can use pipelines to create infrastructure.
Create a pipeline

Create all the pipelines you need to deploy the service or services covered by the application, in whatever ways you want to deploy.

Learn more in the Managing pipelines guide .
Run your pipeline to deploy your service

You can run a pipeline manually, but most pipelines are triggered automatically .
For practice, and to see some sample deployment scenarios, check out our codelabs .

Docs: Google Compute Engine - Bakery

Mon, 01 Jan 0001 00:00:00 +0000

The GCE bakery configuration allows you to set the default network and zone and whether to use the public IP address of the VM used for baking the image.

For example, to set the default zone, add to the rosco-local.yml the following configuration

google:
  enabled: true
  bakery-defaults:
    zone: <zone>
    network: <network>
    networkProjectId: projectId
    subnetwork: subnetwork
    useInternalIp: <true|false>
    templateFile: gce.json
    baseImages:
    - id: unique-bake-id
      shortDescription: shortUI Description (aka precise)
      detailedDescription: Ubuntu precise
      packageType: rpm
      templateFile: gce.json
      osType: <defaults to Linux>
      customRepository: customRepoURL
      isImageFamily: // this is unique to google.

These base images are used to dynamically populate the bake stage UI:

For more examples see the default rosco.yml example file

Docs: Google Groups

Mon, 01 Jan 0001 00:00:00 +0000

Service account setup

In order to access a user’s group membership, we must use the Google Admin Directory API. We will setup a Google Cloud Platform (GCP) service account and grant it access to the Directory API.

Enable the Admin SDK here .
In your Cloud Console , create a service account that will access the G Suite Directory API.
1. Navigate to the “IAM & Admin” –> Service accounts section.
2. Click “Create Service Account.”
3. Give it a name like “spinnaker-fiat.”
4. Select “Furnish a new private key” and select the JSON format.
5. Select “Enable G Suite Domain-wide Delegation.”
6. Click “Create”, which should download the private key for your new service account (see figure below).
7. Transfer this JSON file to a known location in your Spinnaker deployment.
8. You should see your newly created service account in the list, along with “DwD” and an option to “View Client ID”. Click “View Client ID” (see figure below).
9. Note the Client ID displayed. It will be used in the next step.
Give your service account access to the G Suite Directory API in the G Suite Admin console .
1. Navigate to the “Security” Settings page.
2. Under “Advanced settings”, click “Manage API client access.”
3. Enter the Client ID noted above under “Client Name.”
4. Enter https://www.googleapis.com/auth/admin.directory.group.readonly as the API scope.
5. Click “Authorize.”

Configure Fiat

NOTE you’ll want to make sure fiat is enabled in the spinnaker.yml file as it defaults to disabled at this point in time. It’s likely to become mandatory in a future release.

Make sure you’ve configured roles for accounts, as described here . Each role included in the command must match the name of a group in the organization.
With the authorized service account’s credentials in hand, add the following to fiat-local.yml

auth:
  group-membership:
    service: google
    google:
      credentialPath: /mnt/something/sa-creds.json
      adminUsername: emailOfServiceAccount
      domain: googleappsworkspacedomain
      ## this defaults to false.
      expandIndirectGroups: false|true
      ## this is a new parameter to parallel query the google APIs to lookup group information.
      groupLookupConcurrency:  10

Docs: LDAP

Mon, 01 Jan 0001 00:00:00 +0000

Please note that LDAP is flexible enough to offer lots of other options and configuration possibilities. Spinnaker uses the Spring Security libraries, which solve a number of challenges.

Configure fiat

With the LDAP manager credentials and search patterns in hand, add the following to fiat-local.yml

auth:
  group-membership:
    service: ldap
    ldap:
      url: ldaps://something.com
      managerDn: dn=something
      managerPassword: encrypted:secretPlease
      userSearchBase: searchBase
      groupSearchBase: ""
      ## optional, defaults to the uid={-},ou=users
      userDnPattern: uid={0},ou=users
      ## user search fitler defaults to null but could be set like so
      userSearchFilter: (employeeEmail={0})
      ## Defaults to uniqueMember={0}
      groupSearchFilter: (uniqueMember={0})
      ## defaults to cn
      groupRoleAttributes: cn
      groupUserAttributes: nullByDefault
      thresholdToUseGroupMembership: 100
      enablePagingForGroupMembershipQueries: false
      pageSizeForGroupMembershipQueries: 100
      loadUserDNsBatchSize: 100
      userIdAttribute: employeeEmail
      enableDnBasedMultiLoad: false
      cache:
        ## Defaults to false.  Enable to reduce LDAP load
        enabled: true
        expireAfterWriteSeconds: 600

See the ldap configuration code for more information on each setting.

How Fiat determines group membership

The LDAP provider works by querying the LDAP server utilizing a user as set by the manager-dn and manager password and making a query that uses the logged-in user’s username to filter results.

Fiat will use the “bound” account to do the following:

Make a query using a base of groupSearchBase. THIS IS A REQUIRED FIELD. If not set, no roles get queried.
That query uses groupSearchFilter to find the results.
This uses a parameter of the user’s full DN as a filter. This means the ONLY groups shown are those which the user is a member.
For the groups retrieved, get the role names. This uses the groupRoleAttributess attribute (defaults to cn).

How to determine the “Full DN”

Extract the Root DN from the url (ldaps://my.server/a/b/c → a/b/c)

If com.netflix.spinnaker.fiat.roles.ldap.LdapUserRolesProvider log level is at debug, you should see Root DN: <the actual root DN extracted>
If userSearchFilter is provided then:
- Search LDAP:
  - For userSearchBase
  - Using userSearchFilter aka (uid={0})
- Return root DN computed + found user DN
ELSE when userSearchFilter is not provided:
- Make user DN using userDnPattern
- Return root DN computed + user DN

You must provide either a search filter or a DN pattern. In the case below, the user joe would have a full DN of uid=joe,ou=users,dc=mydomain,dc=net.

The search would be rooted at ou=groups,dc=mydomain,dc=net, looking for directory entries that include the attribute uniqueMember=uid=joe,ou=users,dc=mydomain,dc=net, which is the structure for the groupOfUniqueNames group standard.

The groupRoleAttributes is how the group/role name is extracted. For example, all entries that pass the filter will then have the cn (common name) attribute returned.

IF you want to use a username instead of a user DN for group membership, you can specify {1} instead of {0} for the groupSearchFilter parameter.

Source code

To see the internals (can be useful for debugging):

Fiat: LdapUserRolesProvider
Spring Auth Provider: LdapAuthenticationProviderConfigurer
Gate: LdapSsoConfig

Docs: Notifications and Events Guide

Mon, 01 Jan 0001 00:00:00 +0000

Out of the box, Spinnaker allows you to configure the following types of notifications:

This is discussed in the Configuring Notifications section below.

Additionally, Spinnaker allows you to set webhooks for git triggers. See the Setting up Git Triggers in Spinnaker section.

You can also set Spinnaker to stream all its events to a downstream listener. See the Add a Webhook to Spinnaker section.

Spinnaker is also capable of handling cron-based triggers and detect changes in Jenkins builds and Docker images by enabling continuous integration configuration in spinnaker

Last, pub/sub triggering with both custom and default types is available via aws pub/sub configurations or Google pub/sub integration

Configuring notifications

Notification configurations are in echo-local.yml and settings-local.js

Spinnaker baseURL

You need to set the spinnaker.baseUrl configuration value which is used by spinnaker templates. This should point back to the url for your spinnaker’s UI ( deck ) instance. This url is used in notifications to link back to your spinnaker instance.

Email

Email in spinnaker is provided by Spring Boot Mail starter .

The settings for some popular email providers are listed below:

Email Provider	SMTP username	SMTP password	SMTP server address	SMTP port (TLS)	SMTP port (SSL)	SMTP TLS/SSL required
Gsuite/Gmail	Your email address	Your email password	smtp.gmail.com	587	465	yes
Yahoo	Your email address	Your email password	smtp.mail.yahoo.com	587	465	yes
Hotmail/live	Your email address	Your email password	smtp.live.com	587	-	yes
Outlook	Your email address	Your email password	smtp-mail.outlook.com	587	25	yes

The following is an example of using hotmail to send notifications.

echo-local.yml:

mail:
  enabled: true
  from: <myemail>@hotmail.com
spring:
  mail:
    host: smtp.live.com
    username: <myemail>@hotmail.com
    password: hunter2
    port: 587
    properties:
      mail:
        smtp:
          auth: true
          starttls:
            enable: true
        transport:
          protocol: smtp
#	debug: true <- this is useful if you are mucking around with smtp properties

in settings-local.js (deck)

window.spinnakerSettings = window.spinnakerSettings || {};
window.spinnakerSettings.notifications = window.spinnakerSettings.notifications || {};
window.spinnakerSettings.notifications.email = window.spinnakerSettings.notifications.email || {};
window.spinnakerSettings.notifications.email.enabled = true;

Microsoft Teams

Note, teams has changed their APIs. We are unable at this time to send notifications to teams, but would welcome PRs to fix teams integration.

To enable Microsoft Teams support, add the following statement to the echo-local.yml file:

microsoftteams:
  enabled: true

Spinnaker supports sending notifications to Microsoft Teams using Incoming Webhooks.

To create a custom incoming webhook in Teams, follow the instructions provided here: Add an incoming webhook to a Teams channel

When configuring a Microsoft Teams notification, enter the full incoming webhook URL in the Teams Webhook URL text box.

Slack

For slack, you need to create a custom bot user , then get the access token associated with that new bot user. Then…

Add to echo-local.yml

slack:
  botName: SpinnakerNotifications
  enabled: true
  token: ${ECHO_NOTIFICATION_SECRETS_SLACK_BOT_TOKEN}

Note: your users will need to invite the slack bot to private rooms that want to be notified.

Twilio

For Twilio, you need to add your account credentials . Then… add this to your echo-local.yml

twilio:
  enabled: true
  baseUrl: https://api.twilio.com/
  token: token
  account: TWILIO_ACCOUNT_SID
  from: twilioPhoneNumber

CDEvents

To enable CDEvents notification support, add the following statement to the echo-local.yml file:

cdevents:
  enabled: true

Spinnaker supports sending CDEvents to a configured Message Broker . The Message Broker is defined within the notification itself.

For example, Knative Eventing can be configured as a Message Broker to receive CDEvents.

When configuring a CDEvents notification, In the Edit Notification window, specify CDEvents in the Notify via field and enter the full Message Broker URL in the Events Broker URL text box.

To send a specific type of CDEvents, enter the CDEvents Type in the CDEvents Type text box. Different types of CDEvents can be found in the CDEvents Specification .

Note: Currently Spinnaker supports sending Continuous Delivery Core Events

Using notifications

Once notifications have been configured, you can use them to send changes in pipelines and in the manual judgment stage in Spinnaker.

To set up an application-wide notification, go to Application -> Config -> Notifications,

Click on ‘Add Notification’

Enter your notification details.

You can also set Notifications at the Pipeline level ( under configuration ) and at the stage level ( by clicking on the [ ] Send Notifications for this stage checkbox.

Customizing notifications

Spinnaker will send a standard message based on the event that triggered the notification, indicating the stage (if applicable), the pipeline, the application, and the status of the event. It will also include a link to the pipeline.

For stage-level notifications, you can override the message by editing the JSON of the stage, adding a customBody field (for email) or a customMessage field (for Slack). If you are customizing an email notification, you can use Markdown or HTML to customize the format of the email. If you are using Slack, you can use a more limited range of formatting .

For email messages, you can customize the subject by adding a customSubject field.

All standard SpEL expressions will be evaluated prior to sending the notifications, and can be used in the custom notification fields.

There are two special variables available when adding a custom message or body: executionId (the ID of the execution) and link (a fully-formed URL for the pipeline or stage that triggered the notification). To use either of these fields, just wrap them in two curly brackets, e.g. {% raw %}{{link}}{% endraw %}.

An example of a custom email notification: {% raw %}

{
  "customSubject": "Beginning deployment to production (started by: ${trigger.user})",
  "customBody": "*Pipeline parameters:* ${parameters.toString()}\n\n [View the stage]({{link}}) here.",
  "notifications": [
    {
      "address": "spinnakerteam@spinnaker.io",
      "level": "stage",
      "type": "email",
      "when": [
        "stage.starting"
      ],
    }
  ],
  //...
}

{% endraw %}

A custom Slack message: {% raw %}

  {
    "customMessage": "${trigger.user} started a <{{link}}|deploy to production>",
    "notifications": [
      {
        "address": "spinnaker-prod-deploys",
        "level": "stage",
        "type": "slack",
        "when": [
          "stage.starting"
        ],
      }
    ],
    //...
  }

{% endraw %}

Note: the custom fields will apply to all notifications within a stage. If you have different notifications for different events (e.g. a Slack message when the stage starts and when it completes), the custom fields will be applied to both messages.

Setting up Git Triggers in Spinnaker

Instead of listening for events from Github like in the Docker and Jenkins case, direct triggering from git requires a webhook endpoint to be set up in the corresponding VCS.

Github

Using Github as a Trigger

Go to your project’s Webhooks page, for example https://github.com/spinnaker/igor/settings/hooks

Click “Add webhook”

enter http://[spinnakerapi]/webhooks/git/github in the Payload URL, where spinnakerapi is the location of your gate installation.

select “Just the push event”

make it active

Click “Add Webhook” button

Stash

Go to settings / hooks for your repository

Add a Post Receive webhook

Add the following URL: http://[gate url]/webhooks/git/stash, where gate url is the url to your spinnaker api.

You are now ready to receive events from this repository.

BitBucket Cloud

Go to settings / webhooks for your repository

Click ‘Add webhook’

Add a ‘Repository Push’ or ‘Pull Request Merged’ webhook

Add the following URL: http://[gate url]/webhooks/git/bitbucket, where gate url is the url to your spinnaker api.

You are now ready to receive events from this repository.

Add Git Trigger in Pipeline:

Go to Pipeline > Configuration

Select Trigger Type to be Git

Set Repository Type to be github, stash, or bitbucket

Enter your Repository organization / user

Enter your Repository name

Save your pipeline

Notes

You can access the hash of the build via expression ${trigger.hash}

Add a listening webhook to spinnaker

The echo-rest module in spinnaker allows you to set downstream listeners keeping track of Spinnaker events.

Whenever Spinnaker receives an event from Orca, igor, or echo, it will forward these events to the webhooks registered.

A useful pattern is to have an intermediate service that will filter and convert events you care about into the format you expect.

Configuration in echo.yml

rest:
  enabled: true
  endpoints:
    -
      wrap: false
      url: http://listener.com
    -
      wrap: false
      url: http://listener2.com

This will post every event to listener and listener2

Wrapping

If your endpoint expects events of the form

	{ "eventName" : "abc", "payload" : "[event json]"}

you can wrap the event via this configuration:

  -
      wrap: true
      flatten: true
      url: http://listener3
      eventName: spinnaker_events
      fieldName: payload

The flatten setting will simply make the json in content and details ( see below ) into a json String.

Event types

Here is an example event:

{
    "details": {
      "source": "orca",
      "type": "orca:task:complete",
      "created": "1422487582294",
      "organization": null,
      "project": null,
      "application": "asgard",
      "_content_id": null
    },
    "content": {
      "standalone": true,
      "context": {
        "asgName": "asgard-staging-v048",
        "credentials": "test",
        "deploy.account.name": "test",
        "deploy.server.groups": {},
        "kato.last.task.id": {
          "id": "19351"
        },
        "kato.task.id": {
          "id": "19351"
        },
        "kato.tasks": [
          {
            "history": [
            ],
            "id": "19351",
            "resultObjects": []
          }
        ],
        "notification.type": "enableasg",
        "regions": ["us-west-1"],
        "targetop.asg.enableAsg.name": "asgard-staging-v048",
        "targetop.asg.enableAsg.regions": ["us-west-1"],
        "user": "clin@netflix.com",
        "zones": ["us-west-1a", "us-west-1c"]
      },
      "execution": ...
      "executionId": "62ca5574-0629-419a-b9ac-fb873aa165b2",
      "taskName": "f92239a7-b57a-408d-9d72-3a77484e050b.enableAsg.monitorAsg.9568e7e5-3c37-4699-9e93-f62118adc7c6"
    }
  }

Events have details, which will always be the same.

details.type

The type of the event will outline where the event is coming from:

orca:[task type]:[status] - where task type is either ‘pipeline’, ‘stage’ or ’task’ and status is ‘starting’, ‘complete’, ‘failed’
build - from igor Jenkins events
docker - from igor Docker events
git - from git web triggers

details.application

The application involved in the execution, this will only exist for orca: type events.

content.execution

For orca type events, this is the current execution that has the entire pipeline execution.

Docs: Oracle Cloud Infrastructure - Bakery

Mon, 01 Jan 0001 00:00:00 +0000

The Oracle Cloud Infrastructure (OCI) bakery configuration allows for setting the default availability domain, network, and instance shape of the VM used for baking the image. Add the following to rosco-local.yml to enable oracle account baking.

oracle:
  enabled: true
  bakery-defaults:
    availabilityDomain:
    subnetId:
    instanceShape:
    templateFile: oracle.json
    baseImages:
      baseImageId: 
      sshUserName: 
      packageType: rpm

These images are used to dynamically populate the bake stage UI:

For more information, look at the oracle bake code in rosco

Docs: Managed Delivery Overview

Mon, 01 Jan 0001 00:00:00 +0000

What is Managed Delivery?

Managed Delivery is an initiative with two key goals.

Provide native means for Spinnaker users to declaratively manage software delivery.
- With a declarative model, users describe the desired end-state of a deployed application. Spinnaker continually calculates how this desired state maps to cloud resources. When desired state diverges from actual cloud state, Spinnaker figures out how to safely bring the deployed state in-line with the desired.
- This is in stark contrast to the imperative pipeline model, which requires users to define every step of their delivery flow. Pipelines remain a powerful first class feature of Spinnaker that will offer greater flexibility than Managed Delivery for the foreseeable future. However, we feel the declarative approach is preferable for most delivery flows. Good pipeline design requires a depth of Spinnaker and cloud knowledge, while system failures impacting pipeline execution may leave environments in a transitional state without an automatic path back to normalcy. A declarative model solves both issues.
Provide Spinnaker Operators with powerful new means to raise the level of operational abstraction offered to their users.
- By informing how declarative models are resolved, operators can centrally maintain delivery best practices and automate change propagation without knowledge of how product teams have customized delivery pipelines.
- Examples range from offloading decision making about the regions a service should deploy into, to providing means to rapidly redeploy thousands of services running in hundreds of AWS accounts in order to address an OS level security vulnerability.

Contributing

If you are interested in contributing, the Managed Delivery initiative is under the auspices of the Spinnaker-as-Code SIG

Docs: Spinnaker Architecture Overview

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker microservices

Spinnaker is composed of a number of independent microservices:

Deck is the browser-based UI.
Gate is the API gateway.

The Spinnaker UI and all api callers communicate with Spinnaker via Gate.
Orca is the orchestration engine. It handles all ad-hoc operations and pipelines. Read more on the Orca Service Overview .
Clouddriver is responsible for all mutating calls to the cloud providers and for indexing/caching all deployed resources.
Front50 is used to persist the metadata of applications, pipelines, projects and notifications.
Rosco is the bakery. It produces immutable VM images (or image templates) for various cloud providers.

It is used to produce machine images (for example GCE images , AWS AMIs , Azure VM images ). It currently wraps packer , but also supports helm, kustomize and helmfile bakes for kubernetes manifests.
Igor is used to trigger pipelines via continuous integration jobs in systems like Jenkins and Travis CI, and it allows Jenkins/Travis stages to be used in pipelines.
Echo is Spinnaker’s eventing bus.

It supports sending notifications (e.g. Slack, email, SMS), and acts on incoming webhooks from services like Github.
Fiat is Spinnaker’s authorization service.

It is used to query a user’s access permissions for accounts, applications and service accounts.
Kayenta provides automated canary analysis for Spinnaker.
Keel powers Managed Delivery .

System dependencies

This diagram shows which microservices depend on each other. The green boxes represent “external” components, including the Deck UI, a single-page JavaScript application that runs in your browser.

graph TB deck(Deck) --> gate; api(Custom Script/API Caller) --> gate(Gate); gate --> kayenta(Kayenta); gate --> orca(Orca); gate --> clouddriver(Clouddriver); orca --> clouddriver; gate --> rosco(Rosco); orca --> front50; orca --> rosco; gate --> front50(Front50); gate --> fiat(Fiat); orca --> kayenta; clouddriver --> fiat; orca --> fiat; front50 --> fiat; echo(Echo) --> orca; echo --> front50; gate --> echo; gate --> igor(Igor); igor(Igor) --> echo; keel(Keel) --> clouddriver; keel --> orca; keel --> front50; keel --> fiat; keel --> echo; keel --> igor; gate --> keel; classDef default fill:#d8e8ec,stroke:#39546a; linkStyle default stroke:#39546a,stroke-width:1px,fill:none; classDef external fill:#c0d89d,stroke:#39546a; class deck,api external;

In the table below, A filled cell indicates that the system listed in the heading of that column has a dependency on the system listed in the heading of that row. As all Spinnaker services can be run in their own server group, it is typical for services to resolve their dependencies via load balancers or discovery systems (for example Eureka or Consul ).

Port mappings

By default Spinnaker binds ports according to the following table

Service	Port
Clouddriver	7002
Deck	9000
Echo	8089
Fiat	7003
Front50	8080
Gate	8084
Igor	8088
Kayenta	8090
Orca	8083
Rosco	8087
Keel	7010

Docs: Pipeline Permissions

Mon, 01 Jan 0001 00:00:00 +0000

You can use pipeline permissions as an alternative to manually managing Fiat Service Accounts .

Without pipeline permissions, a Spinnaker operator first has to create a Fiat Service account with the correct permissions. A user can then specify the service account as the RunAsUser per automated trigger. Pipeline permissions simplifies this flow – a user only has to specify the set of roles in the pipeline specification based on which a Fiat service account is automatically created and associated with the pipeline.

Enabling pipeline permissions

Pipeline permissions are disabled by default and can be enabled by setting the following flags:

For Orca, add the following to orca-local.yml

tasks:
  useManagedServiceAccounts: true
  ## Recommended to reduce duplicate service accounts by only creating unique service accounts by role set.  This drastically
  ## can improve spinnaker auth sync times by reducing the number of service accounts, particularly with limited sets
  ## of variable service accounts
  useSharedManagedServiceAccounts: true

For Deck, add the following to settings-local.js

window.spinnakerSettings.feature.managedServiceAccounts = true;

Using pipeline permissions

The permissions selector will show up automatically when an automated trigger is added in the pipeline configuration page in the UI. You can add any of the roles that you currently have. Once you add a role to the pipeline, only users who have all of the specified roles can edit the pipeline. This is similar to the behavior of Fiat service accounts .

Migrating from Fiat service accounts

Once pipeline permissions are enabled, the RunAsUser selector will be hidden from Deck. However, any previously configured triggers will continue to use the previously selected service account in order to maintain backwards compatibility. Newly added triggers will use the permissions specified. If you wish to use pipeline permissions for the older triggers and already have permissions specified, edit your pipeline JSON and remove the RunAsUser field from your trigger, or enable the automatic migration (see next section).

Automatic migration

Front50 can automatically migrate all pipelines from using Fiat Service Accounts to use Pipeline Permissions and managed service accounts. The migrator is disabled by default, and can be enabled by adding the following flag to front50-local.yml:

migrations:
  migrateToManagedServiceAccounts: true

This migration will migrate pipelines that have Fiat service accounts set to the new Pipeline Permissions. It will only run on pipelines where roles are not already present and runAsUser is set to a non-managed service account.

The migration job will automatically create the new managed service users, and the new service user will get the same permissions as the manual service user that it replaces.

If a pipeline has multiple triggers with different runAsUser fields set, the new managed service user will get all of the roles of the different manual service users (as you can only have one managed service user per pipeline). This can potentially remove some users ability to edit affected pipelines.

Docs: Plugin Creators Overview

Mon, 01 Jan 0001 00:00:00 +0000

The contents of this page refer to alpha features in Spinnaker 1.19.4.

This means we are working on their stability and usability, as well as possibly adding or changing features. Expect rough edges, and file issues as needed.

Requirements

Spinnaker v2025.3.0+

Taxonomy

Extension Point: Defined by a Spinnaker service, it represents an official contract for which extensions can be written against. Defined by using an org.pf4j.ExtensionPoint interface. Written by Core Contributors.

Extension: A piece of code that implements an Extension Point. Written by Developers.

Plugin: A collection of Extensions that can be bundled together and shipped as a single artifact. Also written by Developers.

Plugin overview

A plugin enables an operator to extend Spinnaker with custom functionality. Use cases include fetching credentials from a custom authorization service, adding a wait stage to a pipeline, updating a Jira ticket, and sending Echo events to third-party tools.

ExtensionPoint plugins

Spinnaker uses the Plugin Framework for Java (PF4J) to indicate an extension point interface to a service. You can create a plugin that implements the methods declared in an extension point. Creating a plugin based on an extension point has a number of advantages:

It’s the easiest - use the @Extension annotation and implement the methods declared in your chosen extension point
Spinnaker loads the plugin in an isolated classpath
It has the least amount of maintenance work
Updates to Spinnaker are not likely to break your plugin

Finding an extension point

An extension point is an interface that extends org.pf4j.ExtensionPoint and is located in the api module of a service. Spinnaker exposes the following extension points:

Orca
- SimpleStage for creating a custom pipeline stage
- [PreconfiguredJobConfigurationProvider]https://github.com/spinnaker/spinnaker/blob/main/orca/orca-api/src/main/java/com/netflix/spinnaker/orca/api/preconfigured/jobs/PreconfiguredJobConfigurationProvider.java) for provisioning preconfigured Job stages
Echo
- EventListener for processing events posted into Echo

Look through the code or ask in the Spinnaker Slack to find extension points not listed here.

Example ExtensionPoint plugin

The pf4jStagePlugin creates a custom pipeline stage that waits a specified number of seconds before signaling success. Consult the Pipeline Stage Plugin Walkthrough for a detailed explanation of this plugin.

Interface (Non-ExtensionPoint) plugins

The second way you can create a plugin is to implement a regular Java interface that you find in a service. Your plugin uses the PF4J @Extension annotation but does not extend org.pf4j.ExtensionPoint.

Advantages:

Spinnaker loads the plugin in an isolated classpath

Disadvantages:

Requires a moderate knowledge of Spinnaker’s architecture and code
Plugin can break if the service’s interface changes

Example Interface plugin

The pf4jPluginWithoutExtensionPoint plugin extends the functionality of Kork’s SecretEngine . SecretEngine is a regular Java interface that does not import any PF4J classes. pf4jPluginWithoutExtensionPoint’s SillySecretEngine implements SecretEngine and uses the @Extension annotation to identify itself as a PF4J plugin. See the plugin project’s README and code for details on how this plugin works.

Spring plugins

When you can’t find an org.pf4j.ExtensionPoint to use or a Java interface to implement, you can create a plugin using Spring. This is should be done as a last resort, since the disadvantages outweigh the advantages.

Advantages:

Full control

Disadvantages:

Requires an expert knowledge of Spinnaker’s architecture and codebase
Requires working knowledge of Spring
High maintenance; plugin can break when Spinnaker dependencies and functionality change

Example Spring plugin

The Spring Example Plugin does not use a PF4J extension point or dependencies. It uses Spring components and was created to test various use cases. See the project for details.

Docs: SAML

Mon, 01 Jan 0001 00:00:00 +0000

The SAML use case is a special one - it’s the only one where a user’s roles cannot be dynamically updated. This is because the user’s roles are sent in the initial authentication handshake between Gate and the SAML Identity Provider (IdP).

IdP Setup

To enable SAML roles, configure your IdP to include group membership in the assertion (not covered - some providers may not offer this option). By default, Gate looks for the memberOf attribute statement, but this can be reconfigured in Gate’s settings.

When Fiat is enabled, SAML groups are automatically pushed to Fiat upon user login and cannot be updated until the user needs to reauthenticate.

Troubleshooting

Docs: Service Accounts

Mon, 01 Jan 0001 00:00:00 +0000

Service Accounts enable the ability for automatically triggered pipelines to modify resources in protected accounts or applications. Practically speaking, this means that a Git commit could trigger a Jenkins build that could then kick off a pipeline to deploy the newly built image in your access-controlled QA environment. The pipeline would run utilizing the permissions of the service account. Service accounts are NOT authenticated accounts but abstractions around permissions for roles.

A simpler approach for many users is pipeline permissions which enable auto creation of service accounts on demand. This is here for those who do NOT want to allow service account creation and require very tight control over trigger execution permissions

Creating service accounts

Service accounts are persistent and configuration merely consists of giving it a name and a set of roles. Therefore, Front50 is the most logical place to configure a service account. There is no UI for creating service accounts at the moment.

Users with all the roles defined in the service account can grant a pipeline “Run as ” but only if the service account itself has EXECUTE access to that app. Users with EXECUTE access to the app can then run the pipeline with additional set of roles (e.g. access to a provider account or some other app).

FRONT50=http://front50.url:8080

curl -X POST \
  -H "Content-type: application/json" \
  -d '{ "name": "sekret-service-account@spinnaker-test.net", "memberOf": ["myApp-prod","myApp-qa"] }' \
  $FRONT50/serviceAccounts

Note: In the above command, myApp-prod and myApp-qa are roles, not application names.

You can see your new service account that just got created:

curl $FRONT50/serviceAccounts

A Fiat sync may be necessary for all affected users to pick up the changes:

FIAT=http://fiat.url:7003

curl -X POST $FIAT/roles/sync

Confirm the new service account has permissions to the resources that you think it should by querying Fiat:

$ curl $FIAT/authorize/myApp-svc-account

Service account roles

The roles you give this service account determine who has access to use it. In order to prevent a privilege escalation vulnerability, only users with every role the service account has may use it.

For example, if user “Batman” has roles foo, bar, and baz and service account “Alfred” has roles foo and bar, then “Batman” has access to use “Alfred” in his pipelines. However, user “Joker”, with role foo, does not have all of the roles “Alfred” has, so he cannot use it.

Using service accounts

With Fiat enabled, you should now see a “Run As User” option in your Trigger configuration. This list contains all of the service accounts you currently have access to.

Upon saving this pipeline, two authorization checks occur:

Does this user have access to this service account? (If using the UI, this should always be the case.)
Does this service account have access to this application?

At pipeline runtime, standard authorization checks against the account and application occur just as if it were a human user.

Deleting service accounts

To delete a service account, use the following DELETE endpoint in Front50:

# Service Account ID is the same as service account name with all lowercase
SERVICE_ACCOUNT_ID="sekret-service-account@spinnaker-test.net"
FRONT50=http://front50.url:8080
curl -X DELETE "$FRONT50/serviceAccounts/$SERVICE_ACCOUNT_ID"

Similar to service account creation, a Fiat sync (curl -X POST $FIAT/roles/sync ) might be necessary for the changes to be immediately reflected.

Docs: API Rate Limiting

Mon, 01 Jan 0001 00:00:00 +0000

If your Spinnaker deployment has API clients interacting with it, enabling and knowing how to operate the API rate limiter can help keep Spinnaker reliable through spikes of heavy traffic or rogue clients.

Intro

The API Rate Limiter currently supports limiting individual authenticated and anonymous principals, bucketing requests into windows that are refreshed every interval as well as global & per-principal learning mode. Both the capacity, as well as the window size, are configurable globally and per-principal.

Some metrics are generated by the rate limiter:

rateLimit.throttling: Counter. Number of enforced throttled requests.
rateLimit.throttlingLearning: Counter. Number of un-enforced throttled requests.
rateLimit.principal.throttled: Counter. A tagged metric recording the number of throttled requests per-principal. Tag key is principal.
rateLimit.principal.remaining: Gauge. A tagged metric recording the number of remaining requests per-principal. Tag key is principal.

Combined with controller.invocations, these metrics can give you a good sense of your on-going rate limiting configuration.

In the case of failures (Redis unavailable) or configuration errors (principal included in both enforcing and ignoring lists), a request will gracefully fallback to learning mode.

Enabling in gate.yml

The rate limiter requires Redis to be available for Gate.

rateLimit:
  enabled: true
  learning: true
  redis:
    enabled: true
  # The rate (in seconds) that the bucket will be replenished
  rateSeconds: 10
  # The number of allowed requests in the window of rateSeconds
  capacity: 500
  # Specific principals can be given different capacities
  capacityByPrincipal:
  - principal: anonymous
    override: 1000
  # Similarly, rateSeconds can be overridden per-principal
  rateSecondsByPrincipal:
  - principal: anonymous
    override: 5
  # A list of principals that are being enforced. Handy for cases where you want
  # to incrementally enable the rate limiter
  enforcing:
  - example@example.com
  # A list of principals that are in learning mode. This can be useful if you
  # want to give some principals unlimited power. Reconsider doing this :)
  ignoring:
  - unmetered-example@example.com

Integrating clients

The rate limiter exposes four HTTP Headers on each request that a client can ingest to play nicely with the rate limiter:

X-RateLimit-Capacity: Int. Total capacity assigned to the principal.
X-RateLimit-Remaining: Int. Total number of requests left before reset.
X-RateLimit-Reset: Date. When the bucket will be replenished.
X-RateLimit-Learning: Bool. Whether or not the rate limiter is enforcing the principal.

Anonymous clients

If your Spinnaker deployment allows anonymous API access, by default all requests will be bucketed into the same rate limit window. With a little trust, you can allow your users to assign themselves their own anonymous rate limit bucket by sending a X-RateLimit-App header, where the value is the name of their application.

Dynamic configuration

Re-deploying gate to respond to real-world events is non-ideal. Redis has the capability to override any configuration defined by the static configuration. First, a list of key patterns:

rateLimit* - All rate limiter-related keys.
rateLimit:* - Principal rate limiter.
rateLimit:learning - Global learning flag.
rateLimit:enforcing - A list of principals being enforced.
rateLimit:ignoring - A list of principals in learning mode.
rateLimit:capacity:* - Per-principal capacity.
rateLimit:rateSeconds:* - Per-principal rateSeconds.

Examples

$ redis-cli

# Set a new capacity
127.0.0.1:6379> set rateLimit:capacity:chaosmonkey@example.com 2000

# Add the chaosmonkey@example.com principal to the enforcing list
127.0.0.1:6379> sadd rateLimit:enforcing chaosmonkey@example.com
(integer) 1

Reference

Code backing the rate limiter can be found in the gate repository .

Docs: AWS EC2 Launch Templates

Mon, 01 Jan 0001 00:00:00 +0000

Launch Template Support must be enabled in order to use the features described in this page. Follow guidelines here to enable the feature set.

Note: The features supported in the UI (Deck) come with tool tips that can help you learn the new features. Consider trying them out in Deck, especially if they are new to you.

Feature Configuration

Once launch templates support is enabled in Clouddriver, a new set of features are unlocked. Some of them are tied to the EC2 launch template directly like IMDSV2 and others require a launch template to be used in order to support features like SpotAllocationStrategy ( MixedInstancesPolicy features).

Review the sections below for details about the features and sample use cases along with Spinnaker API requests.

Launch Template Only Parameters

Review the sections below to determine which features you want to enable in the Spinnaker UI / Deck. Users will see enabled features as options when configuring a Server Group.

Feature	Description	Deck Setting	Clouddriver API Request Parameter	Release Version	Default value in Spinnaker
IPv6	ASGs associate an IPv6 address to their instances.	enableIPv6	associateIPv6Address	v1.21	no default
IMDSv2	Helps mitigate AWS credential theft from the exploitation of SSRF vulnerabilities in web applications. This is only supported by modern SDKs. Learn more from AWS.	enableIMDSv2	requireIMDSv2	v1.21	false
CreditSpecification	The credit option for CPU usage of the instance. Valid for burstable performance EC2 instances only. Learn more from AWS.	enableCpuCredits	unlimitedCpuCredits	v1.24	default false used only when all instance types in request support bursting.

Additional Features

Enabling launch template support also unlocks additional features offered by Amazon EC2 Auto Scaling that facilitate diversified server groups:

Multiple instance types
Multiple purchase options (On-Demand / Spot) with proportion control
Allocation strategies
Capacity rebalancing

Follow AWS recommended best practices for EC2 Spot to improve your experience with EC2 Spot.

Motivation to diversify instances in your Server Group:

Reduce costs by diversifying instances across purchase options and Spot allocation strategies. See AWS docs to learn more about how to use Spot .
Reduce costs and optimize resources with instance weighting.
Enhance availability by deploying your application across multiple instance types running in multiple Availability Zones.
Enhance availability by proactively augmenting your fleet with a new Spot instance from an optimal pool, before a running instance is interrupted by EC2, by enabling capacity rebalance .
Reduce probability of InsufficientInstanceCapacity exceptions with flexible instance configuration.

Using one or more parameters in the table below will automatically create Server Groups with instances distribution configuration specified, and will use AWS defaults for the parameters not specified. Note that a number of these parameters complement each other. So, combining them can greatly enhance your AWS experience.

Category	Feature / Clouddriver API Request Parameter	Description	AWS Default	Clouddriver Release Version	Deck Setting	Deck Release Version
Multiple instance types with instance weighting and priority, with launch template overrides. Properties overridden will replace the same properties in launch template. AWS docs: Instance Weighting	launchTemplateOverridesForInstanceType.instanceType	Instance Type to override	no default	v1.26	Instance Types / LaunchTemplate -> Instance Types	v1.28
launchTemplateOverridesForInstanceType.weightedCapacity	The number of capacity units for the specified instance type in terms of virtual CPUs, memory, storage, throughput, or other relative performance characteristic. The capacity units count toward the desired capacity.	1 i.e. if no weighted capacity is specified, all instance types (large or small) are treated as the same weight.
launchTemplateOverridesForInstanceType.priority	Optional priority for instance type. Lower the number, higher the priority. If unset, the launch template override has the lowest priority. The order of instance types in the list of launch template overrides sent to AWS is set from highest to lowest priority. Valid values: integer > 0.	first to last in list	v1.27
On-Demand AWS docs: Instances Distribution	onDemandAllocationStrategy	Indicates how to allocate instance types to fulfill On-Demand capacity. The only valid value is prioritized. This strategy uses the order of instance types in the launch template overrides to define the launch priority of each instance type. The first instance type in the list is prioritized higher than the last.	prioritized	v1.26	Instance Types -> Instances Distribution / Instances Distribution
onDemandBaseCapacity	The minimum amount of the Server Group's capacity that must be fulfilled by On-Demand Instances. NOTE: If weights are specified for the instance types in the overrides, set the value of OnDemandBaseCapacity in terms of the number of capacity units, and not number of instances.	0
onDemandPercentageAboveBaseCapacity	The percentages of On-Demand Instances and Spot Instances for additional capacity beyond OnDemandBaseCapacity.	100 i.e. only On-Demand instances
Spot AWS docs: * Instance Distribution * ASG Capacity Rebalance * EC2 Instance Rebalance Recommendations	spotAllocationStrategy	Indicates how to allocate instances across Spot Instance pools. 2 strategies: 1) capacity-optimized (recommended): instances launched using Spot pools that are optimally chosen based on the available Spot capacity. 2) lowest-price: instances launched using Spot pools with the lowest price, and evenly allocated across the number of Spot pools specified in spotInstancePools.	lowest-price
spotInstancePools	The number of Spot Instance pools across which to allocate Spot Instances. The Spot pools are determined from the different instance types in the overrides. Only applicable with 'lowest-price' spotAllocationStrategy.	2
spotPrice	The maximum price per unit hour that the user is willing to pay for a Spot Instance.	On-Demand price for the configuration
capacityRebalance	Enable to allow Amazon EC2 Auto Scaling to attempt proactive replacement of Spot Instances in the server group that have received a rebalance recommendation (i.e. at elevated risk of interruption), before they are interrupted by AWS EC2. Note: Enabling this feature could exceed the server group's max capacity for a brief period of time, leading to higher costs. Learn more in AWS docs.	false i.e. disabled	v1.27	Advanced Settings -> capcity rebalance	v1.27

Use Cases & Sample API Requests

Create a Server Group with launch template

After enabling the launch template feature set in Clouddriver and/or Deck, set setLaunchTemplate to true in order to indicate Spinnaker to create your Server Group with an EC2 launch template.

curl -H 'Content-Type: application/json' -d '{ "job": [
  {
    "type": "createServerGroup",
    "cloudProvider": "aws",
    "account": "my-aws-account",
    "application": "myAwsApp",
    "stack": "myStack",
    "credentials": "my-aws-account",
    "subnetType": "public-subnet",
    "availabilityZones": {"us-west-1": ["us-west-1a","us-west-1b","us-west-1c"]},
    "amiName": "ami-12345",
    "capacity": {"desired": 3,"max": 5,"min": 3},
    "iamRole":"MyInstanceProfile",
    "instanceType":"t3.large",
    "setLaunchTemplate": true,
    "requireIMDSv2": true,
    "unlimitedCpuCredits": true
  }], "application": "myAwsApp", "description": "Create New Server Group in cluster myAwsApp"}' -X POST http://my-spinnaker-gate:8084/tasks

Let’s say, the Server Group created was named myAwsApp-myStack-v005. It is backed by EC2 launch template with IMDSv2 enabled and unlimited CPU credits.

Convert a Server Group with launch template to use mixed instances policy, with multiple instance types and capacity weighting

The Spinnaker operation, modifyServerGroupLaunchTemplate/ updateLaunchTemplate also supports updating a Server Group backed by launch template to use mixed instances policy when one or more mixed instances policy parameters listed above is specified.

curl -H 'Content-Type: application/json' -d '{ "job": [ 
  {
    "type": "updateLaunchTemplate",
    "cloudProvider": "aws",
    "account": "my-aws-account",
    "application": "myAwsApp",
    "stack": "myStack",
    "credentials": "my-aws-account",
    "region": "eu-central-1",
    "asgName": "myAwsApp-myStack-v005",
    "launchTemplateOverridesForInstanceType": [
      {"instanceType":"t2.large","weightedCapacity":"1"},
      {"instanceType":"t3.large","weightedCapacity":"1"},
      {"instanceType":"t2.xlarge","weightedCapacity":"2"},
      {"instanceType":"t3.xlarge","weightedCapacity":"2"}] 
  }], "application": "myAwsApp", "description": "Modify Server Group in cluster myAwsApp"}' -X POST http://my-spinnaker-gate:8084/tasks

Note: instance weighting is requested by vCPUs.

AWS docs:

Diversify instances in a Server Group across purchase options (On-Demand and Spot), multiple instance types with priority assignment:

curl -H 'Content-Type: application/json' -d '{ "job": [ 
  {
    "type": "createServerGroup",
    "cloudProvider": "aws",
    "account": "my-aws-account",
    "application": "myAwsApp",
    "stack": "myStack",
    "credentials": "my-aws-account",
    "availabilityZones": {"us-west-1": ["us-west-1a","us-west-1b","us-west-1c"]},
    "amiName": "ami-12345",
    "capacity": {"desired": 3,"max": 5,"min": 3},
    "iamRole":"MyInstanceProfile",
    "instanceType":"m4.large",
    "setLaunchTemplate": true,
    "onDemandBaseCapacity":1,
    "onDemandPercentageAboveBaseCapacity":50,
    "spotAllocationStrategy":"capacity-optimized",
    "launchTemplateOverridesForInstanceType": [
      {"instanceType":"m5.large","weightedCapacity":"1","priority": 2},
      {"instanceType":"m5.xlarge","weightedCapacity":"2","priority": 1}] 
  }], "application": "myAwsApp", "description": "Create New Server Group in cluster myAwsApp"}' -X POST http://my-spinnaker-gate:8084/tasks

See capacity type for instances in Deck:

AWS docs:

Create a Server Group with AWS recommended best practices for EC2 Spot

The best practices followed in the example:

use multiple instance types
use capacity-optimized Spot allocation strategy to indicate AWS to provision instances from the most-available Spot capacity pools
use default maximum price for Spot i.e. On-Demand price
mix in previous generation instance types
use proactive capacity rebalancing (works best with capacity-optimized spotAllocationStrategy)

curl -H 'Content-Type: application/json' -d '{ "job": [ 
  {
    "type": "createServerGroup",
    "cloudProvider": "aws",
    "account": "my-aws-account",
    "application": "myAwsApp",
    "stack": "myStack",
    "credentials": "my-aws-account",
    "subnetType": "public-subnet",
    "availabilityZones": {"us-west-1": ["us-west-1a","us-west-1b","us-west-1c"]},
    "amiName": "ami-12345",
    "capacity": {"desired": 3,"max": 5,"min": 3},
    "iamRole":"MyInstanceProfile",
    "instanceType":"m4.large",
    "setLaunchTemplate": true,
    "onDemandPercentageAboveBaseCapacity":50,
    "onDemandBaseCapacity":1,
    "spotAllocationStrategy":"capacity-optimized",
    "capacityRebalance": true,
    "launchTemplateOverridesForInstanceType": [
      {"instanceType":"m4.large","weightedCapacity":"1"},
      {"instanceType":"m5.large","weightedCapacity":"1"},
      {"instanceType":"m4.xlarge","weightedCapacity":"2"},
      {"instanceType":"m5.xlarge","weightedCapacity":"2"}] 
  }], "application": "myAwsApp", "description": "Create New Server Group in cluster myAwsApp"}' -X POST http://my-spinnaker-gate:8084/tasks

Note: instance weighting requested by vCPUs

AWS docs:

Docs: Caching: Account Name Based Sharding

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Account name based sharding is applicable to the caching function of a clouddriver. In case of HA mode, it is applicable to clouddriver-caching pods only. This feature is added at caching scheduler(both redis & sql backed) level which means it’s applicable to any cloud provider.

This is a configurable feature. When configured, all the caching agents of an account are run by a single caching pod. The logic of identifying the account whose caching agents are to be run in a particular pod is dependent on the name of the account(hashcode) and the number of available caching pods. When the pods are scaled up or down, the agents will resettle to the same or a different pod based on the sharding logic.

Configuration

This feature can be enabled by setting cache-sharding.enabled property to true which by default is false. The below configuration needs to be added to appropriate configuration file (ex: clouddriver-local.yaml):

cache-sharding:
  enabled: true

Additional Details

The implementation details are available at the following PR links.

feat(cats/sql): account name based sharding logic

feat(cats/redis): account name based sharding logic

refactor(cats/sql): made sharding configuration consistent

Docs: Configure a canary

Mon, 01 Jan 0001 00:00:00 +0000

Before you can add a canary stage to a pipeline, you need to configure what the canary consists of, including:

A name by which a canary stage can choose this config
The specific metrics to evaluate, and a logical grouping of those metrics
Default scoring thresholds (which can be overridden in the canary stage)
Optionally, one or more filter templates

Canary configuration is done per Spinnaker application . For each application set up to support canary, you create one or more configs.

Note: By default, all the canary configs you create are visible to all applications. But you can change that .

The configuration you create here is available to canary stages in pipelines, but those stages are defined separately .

Prerequisites

By default, canary is not enabled for new applications. Several things need to happen before you see the Canary tab in Deck:

The person or people setting up Spinnaker for you must set up Canary .

This includes specifying whether all canary configs are available to all applications (the default) or each application can have one or more configurations that are visible to that application only.
In the Application config, activate the Canary option.

Do this separately for all applications that will use automated canary analysis.

Create a canary configuration

You can create as many of these as you like, and when you create a canary stage, you must select a canary configuration to use. Configurations you create within an application are available to all pipelines in that application, but your Spinnaker might be set up so that all configurations are available to all applications.

Hover over the Delivery tab, and select Canary configs.
Select Add configuration.
Provide a Name and Description.

This is the name shown in the stage config when you create a canary stage for your pipeline.
(Optional) select your telemetry provider from the Metric Store dropdown.

If you have only one provider set up, this drop-down is not shown.

The sections below describe how to specify the metrics and the scoring thresholds and weights.

Create metric groups and add metrics

The metrics available depend on the telemetry provider you use. Spinnaker currently supports Stackdriver , Prometheus , Datadog , Signalfx , and New Relic .

Metrics are evaluated even if they’re not added to groups, but if you want to apply the weighting that determines the relative importance of different metrics, you need to add them to groups .

Create any groups you want to organize the metrics into.

Data is evaluated for all metrics, but metrics scores only affect the canary evaluation for metrics that are grouped here.

Click Add Group to create each group you’ll use. Then select the group and click the edit icon to name it.

For example, you might create a group called “cpu” and add a set of CPU-related metrics. Then when you configure a new metric here, you would select “cpu” for the CPU-related metrics you add.
In the Metrics section, select Add Metric.
Select the group to add this metric to.
Give the metric a name.
Specify whether this metric fails when the value deviates too high or too low compared to the baseline.

Or select either, in which case it fails on deviation in either direction.
Optionally, choose a filter template .

This is only available if your Spinnaker is setup with canaries.

Here’s an example:
```
resource.type = "gce_instance" AND
resource.labels.zone = starts_with("${zone}")
```
Identify the specific metric you’re including in the analysis configuration:
- In the Metric Type field type at least 3 characters to populate the field with available metrics.
  
  For example, if you type cpu you get a list of metrics available from your telemetry provider.
Optionally, if your telemetry provider supports aggregation of results, click Group by and enter the metric metadata attribute by which to group and aggregate the data.

For example, when you create a metric you can group its time series by resource or metric label. You can group a time series by zone, for example (resource.zone). This is supported in Stackdriver and Prometheus only.

Metric groups versus grouping metrics

When you create a canary configuration, you create metric groups , and scoring thresholds and weights are applied to groups (rather than to specific metrics). But the grouping described in this step is for aggregating metrics before they’re returned to Kayenta.
Click OK to save this metric.

Your metric is now listed under the specific group you selected for it, and under All.

Add filter templates

If your telemetry provider is Stackdriver, Prometheus, or Datadog, you can add filter templates and then assign each metric a filter template, if you want.

Click Add Template.
Provide a Name.

This is the name by which you can select it when configuring the specific metric.
In the Template field, enter an expression using the FreeMarker template language.

See the interpolation syntax here .

The expression is expanded using the variable bindings specified via the Extended Params in any canary stage that uses this configuration.

These variable bindings are also implicitly available: project, resourceType, scope, location

Advanced metric configuration

Each metric supports additional configuration options in the analysisConfigurations.canary section of the JSON config. These options control how the judge handles the metric:

NaN and missing data

Option	Default	Description
`nanStrategy`	`remove`	How to handle NaN values: `remove` (filter out) or `replace` (replace with 0.0)
`mustHaveData`	`false`	If `true`, missing data causes the metric to fail (`NodataFailMetric`)
`critical`	`false`	If `true`, a High/Low classification fails the entire canary (score = 0)
`muted`	`false`	If `true`, metric is excluded from scoring but still shown in results

Effect size thresholds

Option	Default	Description
`effectSize.measure`	`meanRatio`	Effect size measure: `meanRatio` or `cles`
`effectSize.allowedIncrease`	`1.0`	Ratio threshold for High classification
`effectSize.allowedDecrease`	`1.0`	Ratio threshold for Low classification
`effectSize.criticalIncrease`	`1.0`	Ratio threshold for critical High (requires `critical: true`)
`effectSize.criticalDecrease`	`1.0`	Ratio threshold for critical Low (requires `critical: true`)

Outlier removal

Option	Default	Description
`outliers.strategy`	`keep`	Outlier handling: `keep` or `remove`
`outliers.outlierFactor`	`3.0`	IQR multiplier for outlier detection

Example JSON configuration

{
  "name": "error-rate",
  "analysisConfigurations": {
    "canary": {
      "direction": "increase",
      "nanStrategy": "replace",
      "critical": true,
      "effectSize": {
        "allowedIncrease": 1.1,
        "criticalIncrease": 1.5
      }
    }
  }
}

For details on how these options affect the judge’s behavior, see How canary judgment works .

Edit a configuration

On the Delivery tab, select Canary Configs.

All available existing configs are listed along the left margin. Note that the application that owns a config is shown under the name, in cases where configs are scoped to individual applications.
Select the config you want to edit.

Docs: Configure an application

Mon, 01 Jan 0001 00:00:00 +0000

Once you’ve created an application , you can finish configuring it before adding infrastructure and pipelines.

Start configuring

From within the application, click CONFIG at the top right of the Spinnaker screen.

The application attributes you defined when you created the app are shown at the top.

Note Spinnaker can infer applications based on existing infrastructure. In these cases, there are no application attributes. For applications that you want to use as applications (as described in this guide) you’ll have to add the attributes.
If you want to change any of these attributes, click Edit Application Attributes.

Create pipeline notifications

Spinnaker can send notifications upon completion of certain types of pipeline activity. These notifications are optional.

Click Add Notification Preferences
Select the notification medium:
- email
- SMS
- Slack
Provide the information for the type you chose:
- email address
- Phone number
Choose each pipeline event that will trigger this notification:
- A pipeline started
- A pipeline finished
- A pipeline failed
For each event, enter any custom text you want included in the notification.

For example, for Any pipeline is complete you could enter “Congratulations, your pipeline has finished!” Custom notification text is optional.
Click Update
Repeat these steps to add more notifications, if you want.
Click Edit next to any existing notification to change its preferences, or click Remove to delete it.

Enable or disable application features

You can disable any of the main features available in your application, if you don’t need them. Disabling one of these features merely removes it from the Deck user interface; no existing data is deleted.

Simply de-select any of the features you don’t need:

pipelines
clusters
load balancers
firewalls

Add custom instance links

You can include custom links to the instance details panel, providing shortcuts to information related to the instance, like logs, health, and so on.

Note: Custom instance links don’t currently work for Kubernetes resources.

These links are resolved using the public IP address of the instance, if available, or the private IP otherwise. The default port is 80; if you want to use a different port, begin the path with :.

In the Links section, click Add Section.
Provide text for the new Section heading.
In the Links fields, type the text that you want to display, and the path to the link target.

You can reference these specific instance attributes by wrapping them in curly braces. For example, to include the region where the instance lives, for a GCE or AWS instance, use {region}. These are for the path only, not the link display text.
Click Add Link for each additional link you want displayed in this section.
Click Add Section for any additional sections.
Click Revert to clear your work in progress.

This does not remove any saved custom instance links.
Click Save Changes when you’re finished.

Note that you can set default links across all applications, as described here .

Refresh caches

Spinnaker keeps a local cache of resources. This section of application configuration merely gives you a way to refresh those caches.

Designate a traffic-guard cluster

A traffic guard is a cluster which you have designated as always having at least one active instance.

With traffic guard enabled, if a user or process tries to delete, disable, or resize the server group, Spinnaker will verify the operation will not leave the cluster with no active instances, and fail the operation if it would.

In the Traffic Guards section, click Add Traffic Guard.
Set the following values:

| Field | Required? | Description | | Account | Yes | The account for which you’re setting up the traffic guard| | Region | Yes | The applicable region from the list, or * for all| | Stack | No | The stack to which to apply this traffic guard. Leave blank to apply the guard only to a cluster that has no stack.| | Detail | No | The detail string necessary to identify this cluster via the application-stack-detail naming convention|
Click Save Changes to apply your new traffic guard, or changes to an existing one.

Docs: Create an application

Mon, 01 Jan 0001 00:00:00 +0000

You can’t create a deployment pipeline unless you have an application to put it in. If you’re accessing Spinnaker for the first time and there are no applications available, the first thing you’re going to do is create one.

From any screen, click on the Applications tab.

Note that there might be some applications there already. In some cases, Spinnaker creates applications automatically based on existing infrastructure.

These applications will be marked “unconfigured”; they won’t have the application attributes that are added when you create a new application. These inferred applications are not suitable for putting your pipelines in; go ahead and create a new one.
Click Actions, then Create Application.

Provide the application attributes in the New Application dialog.

Field	Required	Description
Name	Yes	A unique name to identify this application.
Owner Email	Yes	The email address of the owner of this application, within your installation of Spinnaker.
Repo type	No	The platform hosting the code repository for this application. Stash, Bitbucket, or GitHub.
Description	No	Use this text field to describe the application, if necessary.
Consider only cloud provider health	Bool, default=no	If enabled, instance status as reported by the cloud provider is considered sufficient to determine task completion. When disabled, tasks need health status reported by some other health provider (load balancer, discovery service).
Show health override option	Bool, default=no	If enabled, users can toggle previous option per task.
Instance port	No	This field is used to generate links from Spinnaker instance details to a running instance. The instance port can be used or overridden for specific links configured for your application (via the Config screen).
Enable restarting running pipelines	Bool, default=no	If enabled, users can restart pipeline stages while a pipeline is still running. This behavior is not recommended.

Click Create.

Next steps

You now have an application in which to start adding infrastructure and creating pipelines. These application attributes are the minimum configuration, but you will probably want to finish configuring the application .

Docs: Custom CAs for Webhooks

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Webhook stages enable Spinnaker to make HTTP(S) calls to external web services. If the configured webhook URL has the https:// scheme, Spinnaker will use TLS to communicate with the external server. Spinnaker will attempt to validate the certificate presented by the server by building a chain of trust back to a trusted certification authority (CA) and will refuse to connect if the certificate cannot be validated.

By default Spinnaker uses the trust store provided by the JVM as its source of trusted CAs. The default behavior is sufficient for webhooks to public-facing servers where it is possible to build a chain of trust back to a root CA. Internal servers, however, may have certificates issued by a company-specific CA that is not trusted by a root CA. Webhooks to these servers over https:// will fail using the default configuration.

In order to support this latter use case, Spinnaker allows users to supply additional CAs to trust in addition to the default ones. These additional CAs will be used when negotiating connections for outbound webhooks (including preconfigured webhooks) but will not be used for any other connection initiated by Spinnaker. There is no way to specify additional CAs on a per-webhook basis; the additional CAs will apply to all webhooks.

Create a trust store

Create a trust store in Java KeyStore (JKS) format via:

keytool -import -file <path-to-ca-certificate> -alias <name-of-first-ca> -keystore <name-for-keystore>.jks

where <path-to-ca-certificate> is the path to the certificate for the CA you’d like to trust in PEM format, <name-of-first-ca> is an arbitrary alias for that CA, and <name-for-keystore> is the name of a keystore that will be created.

You will be prompted to create a password for the new key store, which you’ll need to supply to Spinnaker in the next step.

After creating the key store with the above command, you can add additional CAs to the keystore by running the same command but supplying a different CA certificate and alias. You’ll be prompted for the keystore password before the new CA can be added. As this trust store will augment the default trust store, you don’t need to add all of the root CAs to this custom trust store; only CAs that are not in the default trust store need to be added.

Configure Spinnaker to use the trust store

Aadd the following can be added to orca-local.yml:

webhook:
  trust:
    enabled: true
    trustStore: <path to trust store in jks format>
    trustStorePassword: <password for trustStore>

Note: Using a secrets manager for these entries is recommended.

Docs: Custom Job Stages

Mon, 01 Jan 0001 00:00:00 +0000

For users of Kubernetes or Titus, Custom Job Stages offer a native solution for enhancing Spinnaker’s functionality with the power of containers. Custom Job Stages enable operators to extend Spinnaker without having to maintain a custom build of Orca.

Note: Custom Job stages are only supported for the Kubernetes and Titus providers.

How it works

At its core, Custom Job Stages use the Run Job stage to start a Task (Titus) or Job (Kubernetes) to perform some type of work. These jobs are defined as configuration for Orca and registered when the application starts. When used in a pipeline, Orca takes this configuration, generates a Run Job stage and then executes it. To support a more native experience, we provide configuration options for things like stage name, description and parameters so that this stage feels less like a Run Job stage and more like an out of the box stage.

When should I use Custom Job Stages?

Custom Job Stages are perfect for when you need to encapsulate a common task so that all users of Spinnaker can benefit and standardize. For example, if you see a large number of users running database migrations in the same way, but manually defining jobs (via Run Job), simply define a Custom Job Stage which captures all the complexity of the job and exposes a subset of customization options.

Some example use cases for Custom Job Stages include:

Building Docker images with Kaniko
Running integration tests
Interacting with external systems
Performing database migrations

Creating a Custom Job Stage

Custom Job stages are configured as part of Orca’s configuration. To enable them, simply add configurations via the job.preconfigured.{cloudProvider} block in orca-local.yml where cloudProvider is the name of the cloud provider which will be used to run the job. For example, if you are using Kubernetes, the value will be kubernetes. The following properties are supported for all providers.

enabled - Determines if stage will be available in the UI.
label - Human readable stage name.
description - Human readable description of what the stage does.
type - Unique key used to determine stage type.
cloudProvider - Cloud provider to use for this stage. Valid values are titus or kubernetes.
credentials - Account name used when executing the job. Must be valid account for cloudProvider.
waitForCompletion - If true, the stage will not be marked complete until the job completes.
parameters - List of parameters presented to the end user.

Custom Job Stages - Kubernetes

The following properties are supported for the kubernetes cloud provider.

account - Account name used when executing the job. Must be valid account for cloudProvider.
application - Name of the application to associate the job with.
manifest - YAML definition for the Kubernetes Job that will be run when the stage is executed.
propertyFile - The container name to capture output data from. Captured output will be available in the pipeline context for use in downstream stages.

Note: If you’re using the slash (/) character in any manifest annotations, you’ll need to use this special syntax to prevent the slash from being dropped when the application reads its configuration. For example, if your annotation key is iam.amazonaws.com/role you’ll need to define it like so: [iam.amazonaws.com/role].

Custom Job Stages - Titus

cluster - Definition of the cluster to be created when the stage is executed.
region - Region in which to run the underlying job.
propertyFile - File to download once the job completes.

Custom Job Stage parameters

Using parameters with the Custom Job Stage enables your users to customize specific fields of the job configuration at runtime. Before the stage is started, user supplied values will replace the values in the stage’s configuration. Parameters support the following properties

name - Name of parameter.
label - Human readable parameter name (shown in the UI).
defaultValue - default value, if not overridden by user.
description - Description of what parameter is used for.
order - Display order of the parameter in the UI.

Example Custom Job Stage

job:
  preconfigured:
    kubernetes:
      - label: Echo
        type: echoPhrase
        description: Echo a configured phrase
        cloudProvider: kubernetes
        account: kubernetes
        credentials: kubernetes
        waitForCompletion: true
        application: k8s2
        parameters:
          - name: PHRASE
            label: Phrase to say
            description: Phrase to be echoed.
            mapping: manifest.spec.template.spec.containers[0].env[0].value
            defaultValue: "Hello world!"
        manifest:
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: say-hello
            namespace: spinnaker
          spec:
            backoffLimit: 0
            template:
              spec:
                restartPolicy: Never
                containers: 
                  - name: hello
                    image: alpine:latest
                    env:
                      - name: PHRASE
                        value: placeholder
                    command: ["echo", $(PHRASE)]

Using a Custom Job Stage

Custom Job Stages will appear as native stages within the Spinnaker UI. Users can add these stages to their pipelines as they would any other stage.

If the stage defines any variables they will be rendered and presented to the user. At this point, the user is free to override the values as they need.

Custom Job stages also take on a native look and feel while executing. The below screen shots show an example of the Echo stage running. You’ll notice that logs are presented to the user for ease of debugging.

Docs: Custom Webhook Stages

Mon, 01 Jan 0001 00:00:00 +0000

Custom webhook stages provide a simple, yet powerful, way of adding custom stages to Spinnaker. These stages are typically used to make quick API calls to an external system as part of a pipeline. Instead of extending the various components through code, users can simply add configuration to Orca for these stages. They appear in the UI as if they were a native stage.

Creating a Custom Webhook Stage

To set up a custom webhook stage, you’ll need to modify the orca-local.yml configuration file. The webhook.preconfigured property allows you to define multiple webhook stages. Here’s an example configuration:

webhook:
  preconfigured:
    - label: GitHub - Update Commit Status
      type: githubStatus
      enabled: true
      description: Update a GitHub Commit Status
      method: POST
      url: https://api.github.com/repos/your-repo/statuses/${parameterValues['gitCommit']}
      customHeaders:
        Authorization:
          - token YOUR_API_TOKEN
      payload: |-
        {
          "state": "${parameterValues['status']}",
          "target_url": "${parameterValues['targetUrl']}",
          "context": "${parameterValues['context']}"
        }        
      parameters:
        - label: Git Commit
          name: gitCommit
          description: The Git commit SHA
          type: string
        - label: Status
          name: status
          description: The state of the status (e.g., success, failure)
          type: string
        - label: Target URL
          name: targetUrl
          description: The URL associated with this status
          type: string
        - label: Context
          name: context
          description: A string label to differentiate this status from others
          type: string

In this configuration:

label: The name displayed in the UI for the stage.
type: A unique identifier for the stage.
enabled: Determines if the stage is available in the UI.
description: A brief explanation of the stage’s purpose.
method: The HTTP method used for the webhook (e.g., GET, POST).
url: The endpoint for the webhook, which can include parameters.
customHeaders: Any headers required for the HTTP request, such as API tokens.
payload: The JSON payload sent with the request.
parameters: Defines user-input fields in the UI, which can be referenced in the url and payload using SpEL expressions.

Note: Once a property is set within a custom webhook stage, users cannot override that setting via the UI.

Configuring Parameters for Custom Webhook Stages

Parameters allow users to input values when configuring the stage in a pipeline. These values can be dynamically accessed using SpEL (Spring Expression Language) expressions. For example, to set up a parameter for a Git commit SHA:

parameters:
  - label: Git Commit
    name: gitCommit
    description: The Git commit SHA
    defaultValue: ''
    type: string

Currently, only the string type is supported for parameters.

Using a Custom Webhook Stage

Once configured, your custom webhook stage will appear in the list of available stages within the Pipeline editor. You can add it to your pipeline like any other stage. If you’ve specified any parameters, they will be rendered and editable as part of the stage configuration. You can also use SpEL to set these values dynamically during pipeline execution.

Useful Custom Webhook Stage Examples

Update GitHub Commit Status

This stage updates a commit status in GitHub as part of a pipeline execution.

label: GitHub - Update Commit Status
type: githubStatus
enabled: true
description: Update a GitHub Commit Status
method: POST
customHeaders:
  Authorization:
    - token YOUR_API_TOKEN
url: https://api.github.com/repos/your-repo/statuses/${parameterValues['gitCommit']}
payload: |-
  {
    "state": "${parameterValues['status']}",
    "target_url": "${parameterValues['targetUrl']}",
    "context": "${parameterValues['context']}"
  }  
parameters:
  - label: Git Commit
    name: gitCommit
    description: The Git commit SHA
    type: string
  - label: Status
    name: status
    type: string
  - label: Target URL
    name: targetUrl
    type: string
  - label: Context
    name: context
    type: string

Create a New Relic Deployment

This stage sends deployment details to New Relic for tracking.

label: New Relic - Create Deployment
type: newRelicDeployment
enabled: true
description: Create a Deployment in New Relic
method: POST
customHeaders:
  X-Api-Key:
    - YOUR_API_KEY
  Content-Type:
    - application/json
url: https://api.newrelic.com/v2/applications/${parameterValues['appId']}/deployments.json
payload: |-
  {
    "deployment": {
      "revision": "${parameterValues['revision']}",
      "user": "${parameterValues['user']}",
      "description": "${parameterValues['description']}"
    }
  }  
parameters:
  - label: Application ID
    name: appId
    description: New Relic Application ID
    type: string
  - label: Revision
    name: revision
    type: string
  - label: User
    name: user
    type: string
  - label: Description
    name: description
    type: string

Testing Webhook Stages

To test your webhook stages before adding them to your pipeline, you can use services like:

Beeceptor – Set up a mock API endpoint to inspect webhook requests.
RequestBin – Capture and analyze HTTP requests to debug your webhooks.

These tools help ensure your webhook stages are correctly configured before deploying them in Spinnaker.

Docs: Delete an application

Mon, 01 Jan 0001 00:00:00 +0000

If you have any server groups in this application, you need to delete them first.
From the Applications screen, open the application you want to delete.
Click CONFIG.
Scroll to the bottom of the application configuration screen, and click Delete Application.

Docs: Echo: Cassandra to In-Memory

Mon, 01 Jan 0001 00:00:00 +0000

Echo’s scheduler can be run completely in-memory. On startup or redeploy, echo will check cron schedules to see if it needs to retroactively execute any missed triggers. This migration only requires configuration changes.

1. Disable Cassandra in echo.yml

spinnaker:
  cassandra:
    enabled: false

2. Enable in-memory backend in echo.yml

spinnaker:
  inMemory:
    enabled: true

3. Enable the scheduler compensation job in echo.yml

scheduler:
  compensationJob:
    enabled: true
    windowMs: 1800000 # optional

The windowMs property dictates how far in the past echo will look to find missed schedules. By default this is 30 minutes.

4. Deploy new Echo

Docs: Create and use filter templates

Mon, 01 Jan 0001 00:00:00 +0000

Filter templates allow you to compose and parameterize advanced queries against your telemetry provider. Parameterized queries are hydrated by values provided in the canary stage .

Things to keep in mind

Each filter template you create for a given canary configuration is available to each individual metric you add.
The query against your metrics is the metric type plus selectors that refine what is returned in the time series.

For Spinnaker canary purposes, the filter template contains only those refining selectors. The metric type is provided by the list of metrics above.
The purpose of the metric filter is to allow you to parameterize what is in these refining selectors, but it is perfectly legal to use literal values in filter templates.

Create a filter template

In the canary configuration , find the Filter Templates section, and click Add Template.
Provide a name.

This name is then populated in the Configure Metric dialog for each individual metric in this config.
In the Template field, enter the filter.

Apply filter templates to metrics

For any metric in this configuration, click the edit icon.
Click the Filter Template field, and select the template you want to apply to time series using this metric.

Provide runtime values for parameterized filters

In the canary stage config , under Extended Params, click Add Field.

This will create a value for one parameter.
Under Key, type a variable you used in a filter template in the config this stage is using.
Add the value you want for that variable. You can use any literal or pipeline expression .

Docs: Front50: Cassandra to Object Store (S3, Azure, or GCS)

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra is no longer an actively maintained or supported persistence store.

A migration to S3, GCS, or AZS is recommended.

1. Create a bucket and folder

Make up a bucket name (or account name if using Azure) that is consistent with the naming policies for the underlying storage service. For purposes of this document, we’ll pick ${USER}-spinnaker since many storage services (including Amazon S3, Google GCS, and Azure AZS) require globally unique names.

Make up a folder name within the bucket. The default is “front50”. Spinnaker will store all the objects within this folder. The folder name will be used for configuration and managed by Spinnaker. You do not need to physically create the folder.

Note that currently only Amazon Simple Storage Service (S3), Google Cloud Storage (GCS), or Azure Storage (AZS) are supported, and they are mutually exclusive. Pick only one. This is independent of where you are actually running Spinnaker.

A. Create the bucket in S3

If you wish to use S3 as the storage service, create the bucket and underlying root folder.

See Amazon’s Documentation for more information.

To enable versioning on an existing bucket, follow these steps.

B. Create the bucket in GCS

If you wish to use GCS as the storage service, Spinnaker can automatically create the bucket (with versioning) for you if it has the right OAuth scopes (Storage Admin). To create it yourself you are best off using the gsutil tool . Using gsutil, also turn on versioning within the bucket.

gsutil mb ${USER}-spinnaker
gsutil versioning set on ${USER}-spinnaker

C. Create the storage account in AZS

If you wish to use Azure Storage, Spinnaker can automatically create the container and root folder (with versioning). Follow instructions here to create a storage account and make sure to note the storage account name and a key.

2. Disable Cassandra in spinnaker-local.yml

services:
  front50:
    cassandra:
      enabled: false

3. Enable the object store in spinnaker-local.yml

A. Enable S3

If you wish to use S3 as the storage service, set the following:

services:
  front50:
    storage_bucket: YOUR_S3_BUCKET_NAME (From Step #1)
    bucket_root: YOUR_S3_FOLDER_NAME (From Step #1)
    s3:
      enabled: true

B. Enable GCS

If you wish to use GCS as the storage service, set the following YAML properties. The default project name is the ${providers.google.primaryCredentials.project} from spinnaker-local.yml. It is needed only if the bucket does not exist and Spinnaker is going to create the bucket for you. The jsonPath is used to convey the credentials to use in order to access GCS. The default is ${providers.google.primaryCredentials.jsonPath} from spinnaker-local.yml however if you created the bucket in a different project (e.g. in the project that is running Spinnaker as opposed to the project that Spinnaker may be managing) then you may need to supply different credentials.

For more information about credentials and obtaining a JSON credentials file, see the discussion on Service Accounts

services:
  front50:
    storage_bucket: YOUR_GCS_BUCKET_NAME (From Step #1)
    bucket_root: YOUR_GCS_FOLDER_NAME (From Step #1)
    gcs:
      enabled: true
      project: SEE NOTE
      jsonPath: SEE NOTE

C. Enable AZS

If you wish to use Azure Storage, set the following:

services:
  front50:
    azs:
      enabled: true
      storageAccountName: YOUR_STORAGE_ACCOUNT_NAME (From Step #1)
      storageAccountKey: YOUR_STORAGE_ACCOUNT_KEY (From Step #1)
      storageContainerName: front50

4. Export existing applications, pipelines, strategies, notifications and projects

Replace “FRONT50_HOSTNAME” and “FRONT50_PORT” and run the following (“localhost” and “8080” are the defaults):

#!/bin/sh

rm applications.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/v2/applications | json_pp > applications.json

rm pipelines.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/pipelines | json_pp > pipelines.json

rm strategies.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/strategies | json_pp > strategies.json

rm notifications.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/notifications | json_pp > notifications.json

rm projects.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/v2/projects | json_pp > projects.json

5. Deploy new Front50

Run sudo restart front50

6. Import Applications, Pipelines, Strategies, Notifications and Projects

Replace “FRONT50_HOSTNAME” and “FRONT50_PORT” and run the following (“localhost” and “8080” are the defaults):

#!/bin/sh

curl -X POST -H "Content-type: application/json" --data-binary @"notifications.json" http://FRONT50_HOSTNAME:FRONT50_PORT/notifications/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"strategies.json" http://FRONT50_HOSTNAME:FRONT50_PORT/strategies/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"pipelines.json" http://FRONT50_HOSTNAME:FRONT50_PORT/pipelines/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"applications.json" http://FRONT50_HOSTNAME:FRONT50_PORT/v2/applications/batch/applications
curl -X POST -H "Content-type: application/json" --data-binary @"projects.json" http://FRONT50_HOSTNAME:FRONT50_PORT/v2/projects/batchUpdate

Docs: Front50: Cassandra to Redis

Mon, 01 Jan 0001 00:00:00 +0000

1. Install Redis

2. Disable Cassandra in front50.yml

cassandra:
  enabled: false

3. Enable Redis in front50.yml

spinnaker:
  redis:
    enabled: true

4. Export existing applications, pipelines, strategies, notifications and projects

#!/bin/sh

rm applications.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/global/applications | json_pp > applications.json

rm pipelines.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/pipelines | json_pp > pipelines.json

rm strategies.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/strategies | json_pp > strategies.json

rm notifications.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/notifications | json_pp > notifications.json

rm projects.json
curl http://FRONT50_HOSTNAME:FRONT50_PORT/v2/projects | json_pp | jq '._embedded.projects' > projects.json

5. Deploy new Front50

6. Import applications, pipelines, strategies, notifications and projects

#!/bin/sh

curl -X POST -H "Content-type: application/json" --data-binary @"notifications.json" http://FRONT50_HOSTNAME:FRONT50_PORT/notifications/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"strategies.json" http://FRONT50_HOSTNAME:FRONT50_PORT/strategies/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"pipelines.json" http://FRONT50_HOSTNAME:FRONT50_PORT/pipelines/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"applications.json" http://FRONT50_HOSTNAME:FRONT50_PORT/global/applications/batchUpdate
curl -X POST -H "Content-type: application/json" --data-binary @"projects.json" http://FRONT50_HOSTNAME:FRONT50_PORT/v2/projects/batchUpdate

Docs: Getting Set Up for Spinnaker Development

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This page describes the steps a Developer should take to fetch Spinnaker’s codebase and get set up to work on it.

Follow the contributing guidelines if you plan to submit your work as a patch to the open source project.

System Requirements

This guide assumes you have access to a machine with a minimum specification of:

At least 16 GB of RAM
At least 6 cores CPU

MacOS

These are some tools but install additional as needed the tools for your needs

homebrew: Not required but useful - can be installed via here
Docker: either docker desktop or rancher. CAUTION: Spinnaker uses TestContainers heavily and you’ll potentially need to adjust settings for test containers which is out of the scope of this guide.
git: Installed usually via Xcode Command Line Tools - often by trying to run git from a terminal
curl: brew install curl - not required explicitly but useful for testing
JDK 17: A JDK is required since we’re building from source. This can be done using sdkman or any other installation tool.
node/npm: (version >=16)
yarn: npm install -g yarn or guide
mysql: docker run -e MYSQL_ROOT_PASSWORD=changeit -it --rm mysql:8 --verbose is a good basic example but will need the databases configured as documented in the storage configuration . Adjust the docker command to persist if you want to keep your data between runs.
redis: docker run -d --rm --name redis -p 0.0.0.0:6379:6739 valkey/valkey:8 as an example.
Set up your cloud provider of choice . Locally these providers would be added to ~/.spinnaker/clouddriver-local.yml files
Add the other sql/redis/other configuration to your ~/.spinnaker/*.yml files for development

Next, clone the spinnaker monorepo . You can now start spinnaker assuming you’ve configured the necessary settings locally for sql/redis.

cd spinnaker
./gradlew run

This starts up ALL the spinnaker services service via a single gradle task. Services can be started independently as needed. Keep in mind that there is a dependency tree for services that will mean that things won’t start unless a dependent service is up and running. Front50 is a common base service dependency for all the other spinnaker services.

Making Changes to Spinnaker

Once you have a working LocalGit deployment you can begin to make changes to the codebase. After you’ve made edits in the code of a service you can see those changes reflected by restarting the services as needed.

Configuring an IDE

IntelliJ

Import the project into IntelliJ:

Select New > Project from Existing Sources
Navigating to the build.gradle file (i.e., ~/dev/spinnaker/build.gradle)

Debugging

Each Java service can be configured to listen for a debugger. To start the JVM in debug mode, set the Java system property DEBUG=true.

The JVM will then listen for a debugger to be attached on a port specific to that service. The service-specific debug ports are as follows:

Service	Port
Gate	8184
Orca	8183
Clouddriver	7102
Front50	8180
Rosco	8187
Igor	8188
Echo	8189

The JVM will not wait for the debugger to be attached before starting a service; the relevant JVM arguments can be seen and modified as needed in the service’s build.gradle file.

Next steps

If you haven’t done so already, read through the Spinnaker Architecture reference to learn more about the individual services’ responsibilities, their dependencies on each other, and their port mappings.
Consider working on one of the issues marked “beginner-friendly” to start learning and contributing to Spinnaker right away.
Sign up for Spinnaker’s Slack community and join the #dev channel to ask questions and get feedback while developing Spinnaker.
Read through the New Stage guide to add new stages and steps
Read about extending spinnaker with CRD handling

Docs: Hiding Stages

Mon, 01 Jan 0001 00:00:00 +0000

Stages that are not provider-specific will be available by default. To hide specific stages from end-users, set the hiddenStages property in Deck’s settings-local.js to a list of the keys of stages you wish to hide. For example, to hide the Gremlin and Travis stages, include the following in settings-local.js:

 window.spinnakerSettings.hiddenStages = ['gremlin', 'travis'];

Docs: Kubernetes CRD Extensions

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker Extension Points for Custom Resource Definitions

At Google, we’ve built extension points for deep CRD integrations within Spinnaker.

This work has allowed us to support the following features within Spinnaker:

Custom models for representing CRDs as spinnakerKinds
Deploying CRDs with custom Spinnaker artifact types
Custom Kubernetes API versions
Custom Spinnaker naming strategies
Per-account, custom Spinnaker UIs that can run alongside the existing Kubernetes UIs

This guide is for developers who want to duplicate this functionality for their CRDs. It also exists as an explanation of certain code paths within Spinnaker which include hooks with no current corresponding open-source implementations.

Developers who want to implement these features will have to build their own layered version of Clouddriver (found in the monorepo) - see Adam Jorden’s blog post - and should be familiar with the Kubernetes provider and writing code for Clouddriver.

Custom Handlers

The central extension point is the KubernetesHandler - defines the relationship between Spinnaker and your Kubernetes kind.

For example, if you wanted to build a Spinnaker integration for your CRD of kind MyCRDKind, you would start with the following handler:

@Component
public class MyCRDHandler extends KubernetesHandler {

  public MyCRDHandler() {
    // Hook point for registering a custom `ArtifactReplacer`
    // for your CRD. During a deploy operation,
    // if an artifact of the type specified in the replacer is present,
    // the artifact will be inserted into the manifest using the
    // strategy described in the replacer.
  }

  @Override
  public KubernetesKind kind() {
    return KubernetesKind.from("MyCRDKind");
  }

  @Override
  public boolean versioned() {
    // If the CRD resource should be versioned - i.e., assigned a sequence
    // v001, v002, etc.
    return false;
  }

  @Override
  public SpinnakerKind spinnakerKind() {
    // The Spinnaker kind that the resource will be represented as in Spinnaker's API and UI.
    return SpinnakerKind.SERVER_GROUPS;
  }

  @Override
  public Status status(KubernetesManifest manifest) {
    // Includes logic for determining whether your CRD manifest is stable.
    // A deploy manifest operation, for example, will block until this
    // method returns a stable status.
  }

  @Override
  public KubernetesV2CachingAgentFactory cachingAgentFactory() {
    return KubernetesCoreCachingAgent::new;
  }
}

Custom Spinnaker Resource Models

You may want to change how their CRD is represented in Spinnaker’s API. By default, a CRD of spinnakerKind serverGroups will be represented with the model class KubernetesServerGroup

You may want to override this representation, for example, if you want to define how your server group’s region is resolved.

To override the default model class, MyCRDHandler should implement ServerGroupHandler (or ServerGroupManagerHandler if your resource is of spinnakerKind serverGroupManagers). You will be responsible for translating raw Spinnaker cache data into a subclass of KubernetesServerGroup.

Custom Kubernetes API Versions

If you want Spinnaker to support custom Kubernetes API versions, subclass KubernetesApiVersion.

For example,

public class MyApiVersion extends KubernetesApiVersion {
  public static MY_BETA_API_VERSION = new MyApiVersion("myApiVersion/v1beta");

  public MyApiVersion(String name) {
    // Base class maintains state.
    super(name);
  }
}

Custom Spinnaker Naming Strategies

To use a custom naming strategy for your CRD, implement NamingStrategy . For example,

@Component
public class MyManifestNamer implements NamingStrategy<KubernetesManifest> {
  @Override
  public String getName() {
    return "myManifestNamingStrategy";
  }

  @Override
  public void applyMoniker(KubernetesManifest manifest, Moniker moniker) {
    // Strategy for applying a Spinnaker `Moniker` to a Kubernetes
    // manifest prior to deployment.
  }

  @Override
  public Moniker deriveMoniker(KubernetesManifest manifest) {
    // Strategy for deriving a Spinnaker `Moniker` from a Kubernetes
    // manifest.
  }
}

This naming strategy can be referenced in a Kubernetes account config. For example:

kubernetes:
  accounts:
    - name: my-kubernetes-account
      namingStrategy: myManifestNamingStrategy

Be careful - this naming strategy will be applied to all manifests manipulated by this account.

Docs: Manage Applications

Mon, 01 Jan 0001 00:00:00 +0000

Overview

spin can manage the whole lifecycle of your application:

$ spin application

Usage:
   application [command]

Aliases:
  application, applications, app

Available Commands:
  delete      Delete the specified application
  get         Get the specified application
  list        List the all applications
  save        Save the provided application

Flags:
  -h, --help   help for application

Global Flags:
      --config string          path to config file (default $HOME/.spin/config)
      --gate-endpoint string   Gate (API server) endpoint (default http://localhost:8084)
  -k, --insecure               ignore certificate errors
      --no-color               disable color (default true)
      --output string          configure output formatting
  -q, --quiet                  squelch non-essential output

Use " application [command] --help" for more information about a command.

The following assumes Spinnaker is running and Gate is listening on http://localhost:8084. If gate is running elsewhere, you can set the Gate endpoint with the global --gate-endpoint flag.

Managing Your Application’s Lifecycle

Create a new application using `save`

$ spin application save --application-name my-app --owner-email someone@example.com --cloud-providers "gce, kubernetes"

Application save succeeded

Applications can also be updated with the save command.

List our Spinnaker applications with `list`

spin application list
[
...
{
  "accounts": "my-account",
  "cloudProviders": "gce,kubernetes",
  "createTs": "1529349914747",
  "email": "jacobkiefer@google.com",
  "instancePort": 80,
  "lastModifiedBy": "anonymous",
  "name": "my-app",
  "platformHealthOnly": true,
  "providerSettings": {
    "gce": {
    "associatePublicIpAddress": true
    }
},
"updateTs": "1529349915014",
"user": "anonymous"
}
...
]

We see our application in the returned list along with the other existing applications.

Retrieve a single application with `get`

spin application get my-app
{
  "attributes": {
    "accounts": "my-account",
    "cloudProviders": "gce,kubernetes",
    "createTs": "1529349914747",
    "email": "jacobkiefer@google.com",
    "instancePort": 80,
    "lastModifiedBy": "anonymous",
    "name": "my-account",
    "platformHealthOnly": true,
    "providerSettings": {
      "gce": {
        "associatePublicIpAddress": true
      }
    },
    "updateTs": "1529349915014",
    "user": "anonymous"
  },
  "clusters": {
    "my-account-cluster": [
      {
        "loadBalancers": [],
        "name": "my-account-gce",
        "provider": "gce",
        "serverGroups": [
          "my-account-gce-v000",
          "my-account-gce-v001"
        ]
      }
    ]
  },
  "name": "my-account"
}

Note that we retrieve not only the application attributes, but also the clusters associated with the application.

Delete a single application with `delete`

When we’re finished with our Spinnaker application, we can delete it.

spin application delete my-app

Docs: Manage Canary Configs

Mon, 01 Jan 0001 00:00:00 +0000

Overview

$ spin canary canary-configs
Usage:
   canary canary-config [command]

Aliases:
  canary-config, canary-configs, cc

Available Commands:
  delete      Delete the provided canary config
  get         Get the canary config with the provided id
  list        List the canary configs
  retro       Retro the provided canary config
  save        Save the provided canary config

Flags:
  -h, --help   help for canary-config

Global Flags:
      --config string            path to config file (default $HOME/.spin/config)
      --default-headers string   configure default headers for gate client as comma separated list (e.g. key1=value1,key2=value2)
      --gate-endpoint string     Gate (API server) endpoint (default http://localhost:8084)
  -k, --insecure                 ignore certificate errors
      --no-color                 disable color (default true)
      --output string            configure output formatting
  -q, --quiet                    squelch non-essential output

Use " canary canary-config [command] --help" for more information about a command.

The following instructions assume Spinnaker is running and Gate is listening on http://localhost:8084. If gate is running elsewhere, you can set the Gate endpoint with the global --gate-endpoint flag.

Create and update canary configs with `save`

$ spin canary canary-configs save --file <path to canary config json>

Note that save accepts the canary config in JSON format. You can quickly export an existing canary config into a valid argument to the --file flag by using the get command.

List canary configs with `list`

$ spin canary canary-config list

[
...
{
  "id": "canaryConfigId",
  "name": "canaryConfigName"
  ...
},
...
]

Retrieve a single canary config with `get`

$ spin canary canary-config get canaryConfigId

{
  "id": "myPipelineTemplate"
  ...
}

Delete a canary config with `delete`

$ spin canary canary-config delete canaryConfigId

Test a canary config with `retro`

The retro command runs a retrospective analysis given a canary config. You must also supply the control and experiment group locators and analysis time window.

$ spin canary canary-configs retro \
  -f <path to canary config json> \
  --control-group app-control-v001 --control-location us-central1 \
  --experiment-group app-experiment-v001 --experiment-location us-central1 \
  --start 2019-09-17T16:27:19.867Z \
  --end 2019-09-17T17:27:19.867Z

Initiating canary execution for supplied canary config
Spawned canary execution with id 01DR9BEP8XTJQDPVFJ41C9MBJ6, polling for completion...
Retrospective canary execution finished, judgement = PASS

Docs: Manage Pipeline Templates

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Note: spin only supports pipeline templates v2.

spin can manage the whole lifecycle of your v2 pipeline templates:

$ spin pipeline-templates

Usage:
   pipeline-template [command]

Aliases:
  pipeline-template, pipeline-templates, pt

Available Commands:
  delete      Delete the provided pipeline template
  get         Get the pipeline template with the provided ID
  list        List the pipeline templates for the provided scopes
  plan        Plan the provided pipeline template config
  save        Save the provided pipeline

Flags:
  -h, --help   help for pipeline-template

Global Flags:
      --config string          path to config file (default $HOME/.spin/config)
      --gate-endpoint string   Gate (API server) endpoint (default http://localhost:8084)
  -k, --insecure               ignore certificate errors
      --no-color               disable color (default true)
      --output string          configure output formatting
  -q, --quiet                  squelch non-essential output

Use " pipeline-template [command] --help" for more information about a command.

The following assumes Spinnaker is running and Gate is listening on http://localhost:8084. If gate is running elsewhere, you can set the Gate endpoint with the global --gate-endpoint flag.

Managing Your Pipeline Template’s Lifecycle

Create and update pipeline templates with `save`

$ spin pipeline-templates save --file <path to pipeline json>

Pipeline template save succeeded

Note that save accepts the pipeline template in JSON format. You can quickly export an existing pipeline template into a valid argument to the --file flag by using the get command.

List pipeline templates for a set of scopes with `list`

spin pipeline-templates list --scopes app1,app2
# Note: --scopes is optional, by default all pipeline templates are global.

[
...
{
  "id": "myPipelineTemplate"
  "pipeline": {...}
  ...
}
...
]

Retrieve a single pipeline template with `get`

spin pipeline-templates get --id myPipelineTemplate
{
  "id": "myPipelineTemplate"
  "pipeline": {...}
}

Visualize a hydrated pipeline with `plan`

spin pipeline-templates plan --file <path to pipeline config>

{
  "application": "my-spinnaker-app"
  "stages": [...] # Evaluated pipeline config based on template config values.
}

Delete a pipeline template with `delete`

spin pipeline-template delete myPipelineTemplate

Pipeline template deleted

Docs: Manage Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Overview

spin can manage the whole lifecycle of your pipeline:

$ spin pipeline

Usage:
   pipeline [command]

Aliases:
  pipeline, pipelines, pi

Available Commands:
  delete      Delete the provided pipeline
  execute     Execute the provided pipeline
  get         Get the pipeline with the provided name from the provided application
  list        List the pipelines for the provided application
  save        Save the provided pipeline

Flags:
  -h, --help   help for pipeline

Global Flags:
      --config string          path to config file (default $HOME/.spin/config)
      --gate-endpoint string   Gate (API server) endpoint (default http://localhost:8084)
  -k, --insecure               ignore certificate errors
      --no-color               disable color (default true)
      --output string          configure output formatting
  -q, --quiet                  squelch non-essential output

Use " pipeline [command] --help" for more information about a command.

The following assumes Spinnaker is running and Gate is listening on http://localhost:8084. If gate is running elsewhere, you can set the Gate endpoint with the global --gate-endpoint flag.

Managing Your Pipeline’s Lifecycle

Create and update pipelines with `save`

$ spin pipeline save --file <path to pipeline json>

Parsed submitted pipeline: <...>

Pipeline save succeeded

Note that save accepts pipeline in JSON format. You can quickly export an existing pipeline into a valid argument to the --file flag by using the get command. You can also export pipeline JSON from the pipeline UI in Deck by clicking Pipeline Actions > Edit as JSON and copying the JSON contents, e.g.

You can also template the pipeline JSON using your favorite templating engine.

List pipelines in an application with `list`

spin pipeline list --application my-app

[
...
{
  "application": "my-app"
  ...
}
...
]

Retrieve a single pipeline with `get`

Get a single pipeline with get:

spin pipeline get --name my-pipeline --application my-app
{
  "application": "my-app"
  "stages": [...]
}

Start a pipeline execution with `execute`

Start a pipeline execution with execute:

spin pipeline execute --name my-pipeline --application my-app

Pipeline execution started

If your pipeline is parameterized, you can submit a JSON-formatted map of the parameters and their values via the --parameter-file flag

{
  "parameter1": "value1",
  "parameter2": "value2",
  ...
}

Delete a pipeline with `delete`

spin pipeline delete --name my-pipeline --application my-app

Pipeline deleted

Docs: Next Release Preview

Mon, 01 Jan 0001 00:00:00 +0000

Please make a pull request to describe any changes you wish to highlight in the next release of Spinnaker. These notes will be prepended to the release changelog.

Coming Soon in Release 2026.3.0

Breaking Changes

AWS V2 SDK migration

The caching agents and code have been moved to the AWS V2 SDK. Any V1 usage via plugins or similar will no longer work going forward.

Angular removed

Angular has been removed from the Spinnaker project. For any plugins or forks, please migrate to React. This is a substantial migration — a big thank you to the contributors, primarily Matt Gogerly.

SAML configuration has moved to Spring configuration

A few properties remain that enable parsing users and translating custom fields, but all content has moved to native Spring SAML configuration. Full migration docs are available .

For a minimal example:

saml:
  enabled: true
  user-attribute-mapping:
    email: email
    roles: memberOf
  # This is defaulted and not required, but you MUST set the ACS URL below
  login-processing-url: /saml/SSO

spring:
  security:
    saml2:
      relyingparty:
        registration:
          SSO:                          # registration ID (arbitrary key, used in URLs, set here to SSO to mimic Spinnaker behavior)
            acs:
              # Keeps older login URLs for now.
              location: "{baseUrl}/saml/SSO"
            entity-id: spinnaker
            assertingparty:
              metadata-uri: https://idp.example.com/saml/metadata
              singlesignon:
                sign-request: true
            signing:
              credentials:
                - private-key-location: /etc/gate/saml/private_key.pem
                  certificate-location: /etc/gate/saml/certificate.pem

Removals and Deprecations

Binaries & core utilities upgraded/removed

Upgrades/removal of old versions — many older libraries and SDKs have been upgraded and/or removed. Some of these (like Helmfile) do not receive as much testing as we’d like, so please verify if you use these libraries.

Kustomize 5 is added. Kustomize 4 is updated to the latest supported version.
Helmfile is upgraded to 1.7.0.
Packer is upgraded to the latest release (1.14).
kubectl binaries used for operations are updated to 1.30. Older versions are removed. You can select a newer version.
aws-iam-authenticator binary is upgraded to the latest supported version.
Base images are upgraded from Alpine 3.20 to 3.24. Ubuntu is moved to the latest release.

Old Kubernetes resource types removed

Very old API specs are removed . Specifically:

extensions/v1beta1
networking.k8s.io/v1beta1

Associated libraries for Kubernetes are also upgraded to a currently supported release. This removes support for any resources using these types — please update your plugins as appropriate. These API versions were removed in Kubernetes 1.22.

Halyard removed

Halyard is removed from the codebase as of this release. For emergency fixes, PRs can be made to the 2026.2.x release branch. Halyard will no longer be released or actively supported, though it may continue to work as we will continue publishing BOMs for now. As of 2027.0.0, we will stop publishing Halyard BOMs.

Kustomize version 3 deprecated — will be removed in 2027.0.0

Kustomize V3 is deprecated. When using Kustomize in a pipeline, this is what’s referenced when you select KUSTOMIZE as the rendering type. Kustomize 5 is being added, and Kustomize 4 will continue to be supported. With 2027.0.0, Kustomize 3 will be removed. Please upgrade your pipelines to Kustomize 4 or 5 before that release.

SQL is the only supported storage for execution data — Redis will be removed in 2027.0.0

In Orca (the execution engine), we are deprecating support for any pipeline storage other than SQL. Please migrate your executions to SQL as soon as possible. Redis-based storage of pipeline execution state will be removed in 2027.0.0. Note: this only impacts storage of pipelines, not the queue system. It is recommended to stay on Redis for the queue system at this time.

Titus deprecated — will be removed in 2027.0.0

Given the lack of contributions, the Titus cloud provider will be removed in an upcoming release. It is marked deprecated as of this release.

SQL is the only supported storage for pipelines/templates — blob storage will be removed in 2027.0.0

Front50 currently supports S3/GCS/etc. storage for pipelines and templates. This will be removed in 2027.0.0. See the instructions on how Netflix migrated to move before that release. All non-SQL storage for Front50 is now marked deprecated.

Features

Configurable timeout on manifest stable checks

Kubernetes manifest stable time — Kubernetes deploys previously had a fixed 30-minute timeout before Deployments were considered “stable.” This timeout is now configurable via a pipeline parameter.

Multi-pipeline runner

Multi-pipeline runner — This plugin enables calling several child pipelines dynamically via a configuration block. For more information and configuration options, see the linked PR.

New scheduler system (Alpha)

Pub/Sub Cloud Account Scheduler — A new scheduler that processes requests for caching and agent operations in order. The current Redis scheduler is non-deterministic about when agents are processed, so users with large numbers of accounts may not get their data processed in a timely manner. This new scheduler exposes endpoints, stores agent state, and surfaces more detailed metrics on agent executions.

To try it out, disable the current Redis scheduler and enable this via:

spring:
  data:
    redis:
      url: redis://valkey:6379

cats:
  pubsub:
    enabled: true
    # Interval between state checks and re-queuing requests for additional processing.
    delayBetweenSchedulerRunsMs: 15000
    # When an agent is removed (e.g. account deletion), it is marked for deletion. This controls when it is actually removed. Time is in minutes. Defaults to 3 hours.
    minutesBeforeDeletingMarkedForDeletion: 180
    # If an agent hasn't been processed for some reason (e.g. lost in the queue), requeue it after this period.
    minutesBeforeReQueueOfAgents: 20
    # Strict concurrency limit on how many agents will run simultaneously. Similar to max-concurrent-agents in other schedulers.
    maxConcurrentAgents: 100
    # Max length of the Redis streams used for picking up agents to run. Set this large if you have many agents.
    # Rough estimate: every AWS account adds ~15 agents per region. 100 accounts × 4 regions × 15 = 6,000 agents minimum.
    # This defaults to 100,000. Note: this value directly affects Redis memory usage.
    streamMaxLength: 100_000

This is an initial step toward using streams to request information rather than polling on a fixed cycle, and enables future alternative agent operation strategies.

Global banner — Admins can create and set banners visible across all Spinnaker applications for global notifications.

Spin CLI API token support

The new API token can be used with the Spin CLI. This is available in current main images.

UI support for adding/removing accounts

Admin-restricted. Adds a new UI panel to add and remove accounts with example payloads.

Fixes

PostgreSQL compression issue

PostgreSQL compression failures — When using PostgreSQL with Orca database compression, failures occurred due to library upgrades. This is now fixed.

Large YAML support

Large YAMLs could fail to load — Following prior snakeyaml upgrades, some parts of Spinnaker did not allow customization of the maximum YAML size (for example, Helm repo indexes). These configurable limits are now universally applied.

Fixed a bug on account APIs

In certain edge cases, newly created accounts would not be read or updated across all pods after creation. This is now fixed.

Docs: Orca: Quality of Service

Mon, 01 Jan 0001 00:00:00 +0000

EXPERIMENTAL: This feature is still in an early adoption / experimental phase. While you can use it today (Orca v6.71.0), Netflix is currently running this in learning mode / judiciously enabling in response to on-call events.

Spinnaker ships with an optional Quality of Service (QoS) module that can be used to manage the amount of active executions running at any given time. By default, this QoS module is disabled, but can be enabled and tuned with a handful of knobs and different strategies. Before we dive into the configuration settings, we’ll go over how QoS works.

How QoS Works

The QoS system is self-contained within Orca’s orca-qos module and operates only on newly created executions (both pipeline and orchestrations). That is to say, the QoS system will not impact running executions: Once an execution has started, it is completely outside of the domain of the QoS module. The rest of this section assumes that QoS is enabled.

When an execution is submitted to Orca (either manually via the API or UI, or through an automated trigger), Orca will first emit a synchronous BeforeExecutionPersist event which the QoS ExecutionBufferActuator is listening on. The behavior of the ExecutionBufferActuator depends firstly on the result of a BufferStateSupplier . The BufferStateSupplier can perform whatever heuristics necessary to determine whether or not any new execution should go through the QoS process. If the BufferStateSupplier returns false, no other QoS actions occur and the execution is started as normal.

In the event BufferStateSupplier returns true, the execution is passed through a chain of ordered BufferPolicy functions. These BufferPolicy functions return a result defining whether or not to BUFFER or ENQUEUE the execution. All BufferPolicy functions must return ENQUEUE, otherwise the execution will be assigned a status of BUFFERED, delaying the initialization of the execution. When an execution is BUFFERED, it will effectively stay in a waiting state until it is unbuffered, which we’ll go over later.

BufferPolicy functions are pluggable and can contain arbitrary logic. For example, one BufferPolicy that is always enabled is EnqueueDeckOrchestrationBufferPolicy , which will always ENQUEUE an execution it is an Orchestration and from the UI. This specific policy forces the ENQUEUE status, even if other policies call for the execution to be buffered; this is done through a force flag that policies can return. An example of other pluggable behavior is determining buffering action based on criticality of the execution: At Netflix we have a custom concept of application criticality, so we can buffer low criticality executions to allow capacity for higher criticality executions.

Once an execution is put into a BUFFERED state, it will remain in that state until the ExecutionPromoter decides to change it to NOT_STARTED and enqueue it for processing. Similar to the ExecutionBufferActuator, the ExecutionPromoter uses an ordered chain of PromotionPolicy functions to determine what buffered executions to promote. Every promotion cycle, the list of buffered executions (called candidates) are passed through the policies, each reducing the list of candidates to a final list of executions that will be promoted. Again, these policies can have arbitrary logic, but by default a naive promotion policy is included that will promote the N oldest executions. This promotion process happens (by default) on a 5-second interval on every Orca instance.

With both BufferPolicy and PromotionPolicy, the results of each function returns a result with a human readable “reason”, which is logged out for each execution that is evaluated so it is easy to trace.

sequenceDiagram participant ExecutionPersister participant ExecutionBufferActuator participant BufferPolicy participant ExecutionPromoter participant PromotionPolicy participant ExecutionLauncher ExecutionPersister->>ExecutionBufferActuator: BeforeExecutionPersistEvent ExecutionBufferActuator->>BufferPolicy: Execution loop Buffer Chain BufferPolicy->BufferPolicy: Evaluate if BUFFERED or ENQUEUED end alt ENQUEUED BufferPolicy->>ExecutionLauncher: Start Execution else BUFFERED Note over BufferPolicy: Set BUFFERED status end note right of ExecutionPromoter: Every n seconds ExecutionPromoter->>PromotionPolicy: All BUFFERED executions "candidates" loop PromoteCycle PromotionPolicy->PromotionPolicy: Reduce candidates end PromotionPolicy->>ExecutionPromoter: Final promotion candidates loop Promote Executions note over ExecutionPromoter: For each promoted execution ExecutionPromoter->>ExecutionPersister: Update Execution status to NOT_STARTED ExecutionPromoter->>ExecutionLauncher: Start Execution end

Note: This is the first implementation of the QoS system, we plan to iterate on this concept and make it more advanced over time. You can read the original proposal to get an idea of a potential roadmap.

Configuration

These configurations are not guaranteed to be fully inclusive of all knobs. A definitive list is available via the codebase.

qos.enabled: Boolean (default false). Global flag controlling whether or not the system is enabled. This flag will not disable the ExecutionPromoter.
qos.learningMode.enabled: Boolean (default true). If enabled, executions will always be ENQUEUED, but log messages & metrics will be emitted saying what the system would have done. This flag has no effect on ExecutionPromoter.
pollers.qos.promoteIntervalMs: Integer (default 5000). The time (in milliseconds) that the promotion process will be run.

BufferPolicy: Naive

The NaiveBufferPolicy will always buffer executions when enabled.

qos.bufferPolicy.naive.enabled: Boolean (default true).

BufferStateSupplier: ActiveExecutions

The ActiveExecutionsBufferStateSupplier will enable/disable the buffering state based on the number of active executions in the system.

qos.bufferingState.supplier must be set to activeExecutions.
qos.bufferingState.activeExecutions.threshold: Integer (default 100). The high threshold of active executions before QoS will start actuating on executions.
pollers.qos.updateStateIntervalMs: Integer (default 5000). The time (in milliseconds) that the function will update its internal record for how many executions are running in the system.

BufferStateSupplier: KillSwitch

The KillSwitchBufferStateSupplier will enable/disable the buffering state based on configuration only. This is handy if you’re evaluating the fundamentals of the QoS system, or you want a break-the-glass operator knob to control QoS.

qos.bufferingState.supplier must be set to killSwitch.
qos.bufferingState.killSwitch.enabled: Boolean (default false). If true, QoS will be enabled.

PromotionPolicy: Naive

The NaivePromotionPolicy will promote N executions every promotion cycle.

qos.promotionPolicy.naive.enabled: Boolean (default true). Whether or not this policy is enabled.
qos.promotionPolicy.naive.size: Integer (default 1). The max number of executions to promote.

Monitoring

qos.executionsBuffered: Counter. The number of executions that have been buffered.
qos.executionsEnqueued: Counter. The number of executions that have been enqueued (e.g. passed through the system and were judged not to be buffered).
qos.actuator.elapsedTime: Timer. The amount of time that is spent passing an execution through all enabled BufferPolicys.
qos.promoter.elapsedTime: Timer. The amount of time that is spent passing an execution through all enabled PromotionPolicys. Since the promoter is run on a static interval, this should usually be a relatively high, yet constant, number.
qos.promoter.executionsPromoted: Counter. The number of executions that have been promoted.

Additional Notes

The QoS system is currently shared-nothing state. Each Orca instance will maintain its own state (aside from configuration) about whether or not it should be buffering executions, or when it should be running pollers.

Docs: Orca: Redis to SQL Migration

Mon, 01 Jan 0001 00:00:00 +0000

If you are not migrating an existing Orca deployment, refer to Orca SQL Setup instead.

Migrate from Redis to SQL

Migrating without downtime from Redis to SQL is a three-step process:

When DualExecutionRepository is running, writes will be routed to either Redis or SQL. Executions will only be migrated to SQL once they’ve completed (either successfully or terminally): This keeps the migration story simple. As such, the migration agents will need to run for awhile. At Netflix, we ran the migration cluster for two weeks, as we had long pipeline executions due to canaries. You may only need to run the migration cluster for an hour.

NOTE: Deploying the migrators as a separate cluster is optional, however the migration process is memory hungry, so you may need to devote more resources to the Orca process.

Enable DualExecutionRepository

Building atop the baseline configuration above, add the following to orca.yml:

executionRepository:
  dual:
    enabled: true
    primaryName: sqlExecutionRepository
    previousName: redisExecutionRepository
  sql:
    enabled: true
  redis:
    enabled: true

Note that both repositories are enabled. Orca will fail to start up if the DualExecutionRepository is misconfigured.

Deploy a Migration Cluster

At Netflix, we deployed orca-main, the cluster that serves our production traffic, as well as orca-main-sqlmigration, which does not receive API traffic nor process the work queue. Its sole purpose is to shovel bits from Redis to SQL.

To perform a deploy a migration cluster, add the following configuration to orca.yml:

---
spring:
  profiles: sqlmigration

pollers:
  orchestrationMigrator:
    enabled: true
    intervalMs: 1800000
  pipelineMigrator:
    enabled: true
    intervalMs: 1800000

queue:
  redis:
    enabled: false

keiko:
  queue:
    enabled: false

You will need to launch this migration cluster with -Dspring.profiles.active=sqlmigration. Spring Profiles allow you to configure a service to startup with different configurations.

Disable DualExecutionRepository

Once all executions have been migrated, you can deploy Orca once again without DualExecutionRepository and delete the migration cluster.

SQL-specific Metrics

Orca emits a few handfuls of metrics that are specific to SQL.

The SqlHealthcheckQueueActivator class, which will disable work queue processing if SQL connectivity goes unhealthy, emits one metric: sql.queueActivator.invocations with a tag of status (disabled or enabled).

The ExecutionRepository will emit a bunch of invocation and timing metrics with the following patterns:

sql.executionRepository.$method.timing
sql.executionRepository.$method.invocations

If you are using the default HikariCP connection pool:

sql.pool.$poolName.connectionAcquiredTiming
sql.pool.$poolName.connectionUsageTiming
sql.pool.$poolName.connectionTimeout
sql.pool.$poolName.idle
sql.pool.$poolName.active
sql.pool.$poolName.total
sql.pool.$poolName.blocked

If you are using the MariaDB driver extension:

sql.pool.$poolName.active
sql.pool.$poolName.idle
sql.pool.$poolName.total
sql.pool.$poolName.blocked

Docs: Orca: Zombie Executions

Mon, 01 Jan 0001 00:00:00 +0000

Aliases: orphaned execution

A zombie Execution is one that has a status in the database of RUNNING but there are no messages in Orca’s work queue or unacked set—the pipeline or task is not doing anything.

Diagnosis

Logs will be emitted regularly for Executions that are currently running in Orca via the QueueProcessor class, which will look similar to the following example. If no logs have been emitted for over 10 minutes for a RUNNING Execution, it is very likely a zombie.

Received message RunTask(executionType=pipeline, executionId=01CT1ST3MBJ9ECPH5JM5HVJARE, application=myapplication, stageId=01CT1ST4P79Y3MPW6FC4H38N3A, taskId=8, taskType=class com.netflix.spinnaker.orca.clouddriver.tasks.instance.WaitForUpInstancesTask)

Metrics & Alerting

Orca can be configured to detect and emit metrics for zombie Executions. This setting is expensive with the RedisExecutionRepository and is disabled by default. You can enable this detection by setting queue.zombieCheck.enabled: true in your Orca configuration.

When enabled, any discovered zombies will be logged out, as well as emitted via a metric:

Found zombie executionType=pipeline application=myapplication executionName=myexample executionId=01CS076X85RX6MWBTQ0VGBF8VX

If you’ve enabled the zombie check, set an alert on the metric queue.zombies, triggering whenever there is a count greater than 0.

Remediation

Rehydrate the Queue

If the Execution is a zombie, there are no messages on the work queue for that Execution. You can attempt to re-hydrate the queue — reissue messages onto the work queue based on the last stored state — using an admin API in Orca , which must be called directly as it is not exposed through Gate. This command can take either a single execution or operate on all executions within a time range. This command will dry-run by default. To actually rehydrate the queue, pass the query parameter dryRun=false.

$ curl -XPOST \
  https://localhost:8083/admin/queue/hydrate?executionId=01CS076X85RX6MWBTQ0VGBF8VX&dryRun=false

This command is best effort and may not be able to rehydrate the Execution, especially if the Execution was zombied while running a non-retryable task.

An example response from the endpoint:

{
  "dryRun": false,
  "executions": {
    "01CS076X85RX6MWBTQ0VGBF8VX": {
      "startTime": 1538679600852,
      "actions": [
        {
          "description": "Task is running and is retryable",
          "message": {
            "kind": "runTask",
            "executionType": "PIPELINE",
            "executionId": "01CS076X85RX6MWBTQ0VGBF8VX",
            "application": "myapplication",
            "stageId": "01CS076X8501MNAD2ZTJ4ST2TM",
            "taskId": "1",
            "taskType": "com.netflix.spinnaker.orca.echo.pipeline.ManualJudgmentStage$WaitForManualJudgmentTask",
            "attributes": [],
            "ackTimeoutMs": 600000
          },
          "context": {
            "stageId": "01CS076X8501MNAD2ZTJ4ST2TM",
            "stageType": "manualJudgment",
            "stageStartTime": 1538682406227,
            "taskId": "1",
            "taskType": "waitForJudgment",
            "taskStartTime": 1538682406242
          }
        },
        {
          "description": "Task is running but is not retryable",
          "context": {
            "stageId": "01CS076X85ECXHF3FRWZBTQ359",
            "stageType": "createProperty",
            "stageStartTime": 1538681485559,
            "taskId": "3",
            "taskType": "monitorProperties",
            "taskStartTime": 1538681546116
          }
        }
      ],
      "canApply": false
    }
  }
}

For each Execution, a final action summary is provided canApply. If any part of an Execution cannot be re-hydrated, the entire Execution will be skipped.

Cancel the Execution

If the Execution cannot be rehydrated, it will need to be canceled. You can cancel the Execution via the UI or force cancellation via an Orca admin API:

PUT /admin/forceCancelExecution?executionId=01CS076X85RX6MWBTQ0VGBF8VX&executionType=PIPELINE

Known Causes

Zombie Executions can occur due to a loss of the Redis instance backing the Orca queue, or in prolonged unreliable networks. If you’re using the RedisExecutionRepository, it’s likely that when you lose the Redis backing the queue, you’re also losing all running Executions. However, while using the SQL backend for Orca , losing the Redis means you’re only losing the in-flight work queue state, not the state of the pipelines. In such a scenario, once Redis has been restored it will not have any messages to process and existing RUNNING Executions will sit unprocessed.

Docs: Releasing A Patch

Mon, 01 Jan 0001 00:00:00 +0000

Say you’ve written a patch, and it’s been merged into Spinnaker. First off, thanks for helping the project! Odds are, you want to deploy this patch to the Spinnaker you manage. You have a few options available:

Wait for the non-patch release

Any time we release a new minor version of Spinnaker (e.g. 1.16.0 or 1.17.0), we include all commits merged into main for each service. We do this on a quarterly basis by default.

Release branch patch criteria

In order to be considered safe to merge into a release branch, your patch must:

Fix a documented regression in a supported version of Spinnaker . This means that the currently broken functionality must have worked as expected in a previous version of Spinnaker. If the regression is not already documented in a GitHub issue, please create one. Describe the difference between the expected and observed behavior, and include links to the commit that introduced the regression. Indicate to which releases you would like your fix to be backported.
Include tests validating the regression and the fix. The first commit of your patch pull request should add test coverage that demonstrates the existence of the bug and exercises all code paths potentially impacted by your fix. Subsequent commits should fix the bug and update the tests you just added. If your fix is so complex as to make complete tests coverage impossible, it is not a good candidate for merging into a release branch.

These criteria do not apply to security vulnerability patches, which may be merged into release branches at the discretion of the Security SIG and release manager.

Merge into the release branch

If your patch meets the cherry-pick criteria , you can request that your patch be merged into a release branch. Every minor release of Spinnaker has its own release branch. For example, all Spinnaker 2026.0.x releases (2026.0.1, 2026.0.2, etc.) are built from the release-2026.0.x release branch. To get your patch into 2026.0.x, it must be cherry-picked onto that release branch.

After you’ve created a pull request for a fix that you want backported to a release branch, add a comment that includes the following:

@spinnakerbot add-label backport-candidate

Release managers will audit all PRs with the backport-candidate label weekly. Please make sure your pull request description makes it easy for the release manager to evaluate whether your patch meets the release branch patch criteria.

Navigate to GitHub, and create a PR as you would normally, but make sure that your “base” is set to the release branch in the upstream repository as shown below:

{% include figure image_path="./patch.png" %}

Once this PR is merged, your patch should be released in the next few days.

Docs: Sharding Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Intro

This document shows you how to shard traffic to different Spinnaker services based upon configured criteria. The general pattern is to define a selector class in your configuration. Endpoints will then be selected based upon the criteria specified in the selectors.

As an example, Netflix creates read-only shards for Clouddriver to better manage requests.

Selectors exist at these levels:

Application
Execution type (i.e, Pipeline vs Orchestration)
Origin
Authenticated User
Deployment account
Cloud Provider

You will need to modify your Spinnaker deployment pipelines to ensure the infrastructure for each shard is correctly created, otherwise traffic could be sent to non-existent shards.

If no selector is specified, the default endpoint will be used.

Additionally, there is a special dynamicEndpoints configuration in gate.yml to send all requests from Deck to that particular shard.

Sharding Orca Requests

In gate.yml

services:
  orca:
    shards:
      baseUrls:
        - baseUrl: https://orca.example.com
        - baseUrl: https://orca-shard1.example.com
          priority: 10
          config:
            selectorClass: com.netflix.spinnaker.kork.web.selector.ByApplicationServiceSelector
            applicationPattern: xxxxyyyapp |demo.*xxxxyyyy

Clouddriver Read-only Shards

gate.yml

services:
  clouddriver:
    baseUrl: https://clouddriver-readonly.example.com
    config:
      dynamicEndpoints:
        deck: https://clouddriver-readonly-deck.example.com

orca.yml

clouddriver:
  readonly:
    baseUrls:
    - baseUrl: https://clouddriver-readonly-orca-1.example.com
      priority: 10
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByExecutionTypeServiceSelector
        executionTypes:
          - orchestration
    - baseUrl: https://clouddriver-readonly-orca-2.example.com
      priority: 20
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByApplicationServiceSelector
        applicationPattern: app1|.*app2.*
    - baseUrl: https://clouddriver-readonly-orca-3.example.com
      priority: 30
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByOriginServiceSelector
        origin: deck
        executionTypes:
          - orchestration
    - baseUrl: https://clouddriver-readonly-orca-4.example.com
      priority: 50
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByAuthenticatedUserServiceSelector
        users:
          - horseman.*
          - bojack.*
    - baseUrl: https://clouddriver-readonly-orca-5.example.com

Clouddriver Write-only Shards

orca.yml

clouddriver:
  writeonly:
    baseUrls:
    - baseUrl: https://clouddriver-write-kubernetes.example.com
      priority: 10
      config:
        selectorClass: com.netflix.spinnaker.kork.web.selector.ByCloudProviderServiceSelector
        cloudProviders: 
          - kubernetes
    - baseUrl: https://clouddriver-write-cloudfoundry.example.com
      priority: 20
      config:
        selectorClass: com.netflix.spinnaker.kork.web.selector.ByCloudProviderServiceSelector
        cloudProviders: 
          - cloudfoundry
    - baseUrl: https://clouddriver-write-heavy-account.example.com
      priority: 30
      config:
        selectorClass: com.netflix.spinnaker.kork.web.selector.ByAccountServiceSelector
        accountPattern: heavyUsageAccount.*|.*app2.*
    - baseUrl: https://clouddriver-write-heavy-users.example.com
      priority: 40
      config:
        selectorClass: com.netflix.spinnaker.orca.clouddriver.config.ByAuthenticatedUserServiceSelector
        users:
          - horseman.*
          - bojack.*
    - baseUrl: https://clouddriver-write.example.com

Docs: Usage Statistics

Mon, 01 Jan 0001 00:00:00 +0000

If enabled , Spinnaker collects data about how the tool is being used. This data is anonymized (before being sent across the internet) and aggregated.

Why is data being collected?

The project maintainers have few signals about how the product is actually being used in the real world.

Is new feature X having any traction with users? Can we remove feature Y because it’s hard to maintain?

Questions like these are difficult to answer without at least some automated data collection, and ultimately that leads to a product that becomes slow, bloated, and littered with failed experimental code that nobody is confident about deleting.

What data is being collected?

Spinnaker’s telemetry module collects the following bits of data:

Spinnaker top-level version
A unique instance ID (a randomly generated ULID), SHA-256 hashed when sent over the wire
Application name, SHA-256 hashed and salted with the instance ID
Each execution’s:
- ID, SHA-256 hashed
- Final status
- Type
- Trigger
- Each stage’s:
  - Final status
  - Type
  - Cloud Provider

Spinnaker’s telemetry project has been reviewed and approved as compliant with the Linux Foundation’s Telemetry Data Collection and Usage Policy .

How is the data collected?

The above payload is sent to https://stats.spinnaker.io/log upon the completion of any pipeline or ad-hoc UI operation. Telemetry is enabled by default. Spinnaker administrators can turn off data collection by adding the following config to echo

stats:
  enabled: false

and redeploying.

What are we doing with this data?

We’re crunching the numbers and giving them back to the community in the form of dashboards (below) and access to the raw data by requesting access here (a Google Account is required in order to access the data via Big Query).

Dashboards

Docs: Writing a New Stage

Mon, 01 Jan 0001 00:00:00 +0000

Overview

To create a new stage, you need to make changes in the monorepo to orca to implement the logic of the stage, and the front-end changes in deck to implement the UI. Depending on what the stage does, you may need to implement new cloud provider-specific logic in clouddriver and/or expose new APIs in gate as well.

This doc currently only covers the backend changes made to orca.

Backend (orca)

For the backend, you need to define:

A stage class
One or more task classes associated with the stage

Stage class

A stage class must implement the com.netflix.spinnaker.orca.api.pipeline.graph.StageDefinitionBuilder interface. Note there are several abstract classes that have extended this with things like expression aware functionality that are usable.

For providing additional functionality, it can also implement other interfaces:

A CancellableStage can be cancelled.
A RestartableStage can be restarted.
A CloudProviderAware stage exposes information about the cloud provider.
An AuthenticatedStage can perform custom authentication.

Here’s an example based on a stage that is used internally at Netflix to integrate with the Chaos Automation Platform (ChAP).

The stage is composed of two tasks:

beginChap starts a new ChAP run
monitorChap waits for the ChAP run to finish

package com.netflix.spinnaker.orca.pipeline;

import com.google.common.collect.ImmutableMap;
import com.netflix.spinnaker.orca.CancellableStage;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.chap.tasks.BeginChapTask;
import com.netflix.spinnaker.orca.chap.tasks.MonitorChapTask;
import com.netflix.spinnaker.orca.pipeline.model.Execution;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
public class ChapStage implements StageDefinitionBuilder, CancellableStage {

  @Autowired
  public ChapService chapService;

  @Override
  public <T extends Execution<T>> void taskGraph(Stage<T> stage, TaskNode.Builder builder {
    builder
      .withTask("beginChap", BeginChapTask.class)
      .withTask("monitorChap", MonitorChapTask.class);
  }

  @Override
  public CancellableStage.Result cancel(Stage stage) {
    Run run = (Run) stage.getContext().get("run");
    if (run != null) {
      Run latestDetails = chapService.cancelChap(run.id.toString(), "");
      return new CancellableStage.Result(stage, ImmutableMap.of("run", latestDetails));
    }

    return null;
  }
}

Task classes

A task class must implement a com.netflix.spinnaker.orca.api.pipeline.Task , or an interface that extends it, such as:

A RetryableTask can be retried if it fails.
A PreconditionTask defines preconditions that the task will enforce.

To communicate that a task failed, throw a RuntimeException.

In our example, the ChapStage consists of two tasks:

BeginChapTask
MonitorChapTask

BeginChapTask

The BeginChapTask:

retrieves the testCaseId specified by the user during the task configuration stage
calls the ChAP service via REST API call (see ChapService.startChap)
returns a DefaultTaskResult, passing it the response from the ChAP REST API call

package com.netflix.spinnaker.orca.chap.tasks;

import com.netflix.spinnaker.orca.*;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Component
public class BeginChapTask implements RetryableTask {

  @Override
  public TaskResult execute(Stage stage) {
    Map<String, Object> ctx = stage.getContext();
    Object testCaseId = ctx.get("testCaseId");

    if(testCaseId == null || !(testCaseId instanceof String)) {
      throw new RuntimeException("Cannot begin ChAP experiment without a testCaseId.");
    }

    Map<String, Object> params = new HashMap<>();
    params.put("testCaseId", testCaseId);
    Run chapRun = chapService.startChap(params);

    Map<String, Object> map = new HashMap<>();
    map.put("run", chapRun);
    return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
  }

  public ChapService getChapService() {
    return chapService;
  }

  public void setChapService(ChapService chapService) {
    this.chapService = chapService;
  }

  @Autowired
  private ChapService chapService;

  @Override
  public long getBackoffPeriod() {
    return TimeUnit.SECONDS.toMillis(5);
  }

  @Override
  public long getTimeout() {
    return TimeUnit.MINUTES.toMillis(1);
  }
}

MonitorChapTask

The MonitorChapTask polls the ChAP service for the status of the ChAP run.

package com.netflix.spinnaker.orca.chap.tasks;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.netflix.spinnaker.orca.DefaultTaskResult;
import com.netflix.spinnaker.orca.ExecutionStatus;
import com.netflix.spinnaker.orca.RetryableTask;
import com.netflix.spinnaker.orca.TaskResult;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.orca.chap.Run;
import com.netflix.spinnaker.orca.pipeline.model.Stage;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.TimeUnit;

@Component
public class MonitorChapTask implements RetryableTask {

  @Autowired
  private ObjectMapper objectMapper;

  @Autowired
  public ChapService chapService;

  @Override
  public TaskResult execute(Stage stage) {
    Map<String, Object> ctx = stage.getContext();

    Run run = objectMapper.convertValue(ctx.get("run"), Run.class);

    if (run == null) {
      throw new RuntimeException("Cannot monitor Chap task without a valid Run in the context.");
    }

    Run latestDetails = chapService.getChap(run.id.toString());

    Map<String, Object> map = new HashMap<>();
    map.put("run", latestDetails);

    if(latestDetails.outcome == Run.Outcome.PASSED){
      return new DefaultTaskResult(ExecutionStatus.SUCCEEDED, map);
    }

    ExecutionStatus status;

    switch (latestDetails.state) {
      case COMPLETED:
        //workflow is complete, but the outcome didnt pass, consider this a failure.
      case FAILED:
        throw new RuntimeException("ChAP experiment failed.");
      case CANCELLED:
        status = ExecutionStatus.CANCELED;
        break;
      default:
        status = ExecutionStatus.RUNNING;
        break;
    }


    return new DefaultTaskResult(status, map);
  }

  public ChapService getChapService() {
    return chapService;
  }

  public void setChapService(ChapService chapService) {
    this.chapService = chapService;
  }

  @Override
  public long getBackoffPeriod() {
    return TimeUnit.MINUTES.toMillis(1);
  }

  @Override
  public long getTimeout() {
    return TimeUnit.DAYS.toMillis(1);
  }

  public ObjectMapper getObjectMapper() {
    return objectMapper;
  }

  public void setObjectMapper(ObjectMapper objectMapper) {
    this.objectMapper = objectMapper;
  }
}

Other classes used

The details of the com.netflix.spinnaker.orca.chap.Run class and com.netflix.spinnaker.orca.chap.ChapService interface aren’t directly relevant to learning how to write a Spinnaker stage, but for completeness, here’s what those look like for this case:

Run

The Run class is a Java object that is deserialized from JSON.

package com.netflix.spinnaker.orca.chap;

import com.fasterxml.jackson.annotation.JsonAnyGetter;
import com.fasterxml.jackson.annotation.JsonAnySetter;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;

import java.util.HashMap;
import java.util.Map;
import java.util.UUID;

@JsonIgnoreProperties(ignoreUnknown = true)
public class Run {

  public UUID id;
  // Other properties not shown
  // ...

  // Support arbitrary properties without needing to define them explicitly
  public Map<String, Object> properties = new HashMap<>();

  @JsonAnySetter
  public void set(String fieldName, Object value) {
    this.properties.put(fieldName, value);
  }

  @JsonAnyGetter
  public Object get(String fieldName) {
    return this.properties.get(fieldName);
  }
}

ChapService

The ChapService defines a REST client API for talking to the ChAP service. It uses the Retrofit library.

package com.netflix.spinnaker.orca.chap;

import retrofit.http.Body;
import retrofit.http.GET;
import retrofit.http.POST;
import retrofit.http.Path;

import java.util.Map;

public interface ChapService {
  @POST("/v1/runs")
  Run startChap(@Body Map params);

  @GET("/v1/runs/{id}")
  Run getChap(@Path("id") String id);

  @POST("/v1/runs/{id}/stop")
  Run cancelChap(@Path("id") String id, @Body String body);
}

ChapConfig

To implement the ChapService interface, we do not define a class that extends the interface. Instead, we define a class named ChapConfig with a Spring @Configuration annotation. Note that this implementation uses the chap.baseUrl configuration value that is defined in a separate Spinnaker configuration file.

package com.netflix.spinnaker.config;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.netflix.spinnaker.orca.chap.ChapService;
import com.netflix.spinnaker.retrofit.Slf4jRetrofitLogger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import retrofit.Endpoint;
import retrofit.RestAdapter;
import retrofit.client.Client;
import retrofit.converter.JacksonConverter;

import static retrofit.Endpoints.newFixedEndpoint;

@Configuration
@ComponentScan({
  "com.netflix.spinnaker.orca.chap.pipeline",
  "com.netflix.spinnaker.orca.chap.tasks"
})
@ConditionalOnProperty(value = "chap.baseUrl")
public class ChapConfig {

  @Bean
  Endpoint chapEndpoint(@Value("${chap.baseUrl}") String chapBaseUrl) {
    return newFixedEndpoint(chapBaseUrl);
  }

  @Bean
  ChapService chapService(Endpoint chapEndpoint,
                          Client retrofitClient,
                          RestAdapter.LogLevel retrofitLogLevel,
                          ObjectMapper objectMapper) {
    return new RestAdapter.Builder()
      .setEndpoint(chapEndpoint)
      .setClient(retrofitClient)
      .setLogLevel(retrofitLogLevel)
      .setLog(new Slf4jRetrofitLogger(ChapService.class))
      .setConverter(new JacksonConverter(objectMapper))
      .build()
      .create(ChapService.class);
  }
}

Docs: Getting Started as a Content Contributor

Mon, 01 Jan 0001 00:00:00 +0000

Originally published on the Spinnaker Community Blog

Hi! Nikema here. I co-lead the Spinnaker Contributor Experience Special Interest Group (SIG) with Adetokunbo Ige. Since stepping into this role, I’ve wanted to write a post to explicitly welcome non-code contributions to the project. This is meant to be a living document, and I am open to feedback and suggestions. Let’s work together to make open source contributing a little less intimidating.

I will be loosely referring to contributing to Spinnaker, but I intend to make this a starting point for anyone new to content contributing.

What You Need to Get Started

Prerequisites

Familiarity with Git and GitHub

The Spinnaker project lives on GitHub. It’s helpful to know how to navigate the platform and complete common tasks. Later in the post, I’ll list some ideas for non-code contributions. Not everything you do as a community member needs to go through GitHub, but it’s a good skill to have and to practice.

If you aren’t familiar with GitHub, Kent C. Dodds has a free community course on egghead.io. He’s a great teacher so this is an awesome place to start.

https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github
An interest in an open source project

My suggestion is to find a project that is exciting to you or solving a problem you have experienced. You may also want to work backwards from finding a great community. Good people can make the most challenging work enjoyable.
Willingness to participate in a community

You’ll have to interact with people as you are making your contributions. There might be times when you have to discuss an issue or Pull Request (PR). Don’t be afraid but keep in mind that you don’t have to put up with toxicity or dismissive attitudes. If people are jerks to you, find a better, more inclusive project to contribute to.

Helpful Resources

Ways to Contribute Content Without Coding

If there’s one point I’d like to get across, it’s that you don’t have to be an expert in the technology to make awesome contributions. In this post, I’ll provide a short list of ideas for creating content for project contributions.

Participate in community discussions

Join the community forum (in our case, it’s Slack) and introduce yourself. Ask questions. Find out what the project needs and how you may be able to help. Make comments and bring yourself to the community.

Learn in public

It’s okay to be a beginner and content from a newbie perspective is incredibly valuable. Once you are familiar enough, it’s hard to remember the things that were challenging or blocking for you when you were just starting out. The more perspectives we have documented, the more accessible our projects are to people within a range of experience levels.

Add your perspective to existing content

View talks, tutorials, and workshops and add your perspective. You could talk about it on video or write a blog post.

What did you learn? What was hard to understand? Did you get stuck?

Again, it’s super-helpful to do this. If you got stuck and were able to resolve an issue, sharing that with the community will help others avoid the problem and could eliminate the problem for future users.

Like (or clap), comment, subscribe, share.

Organize or join local or virtual meetups around a project

Meet up with others. There is so much potential for magic ✨ to happen when we get together.

Give a talk on your experience

Your experience matters. Your unique voice is a value add to the community and no one can provide it but you.

Publish interviews of community members and maintainers

Learn from others through interviews; share them with the community.

Start a study group and document your progress

The journey from newbie to knowledgeable is much more fun with friends. Get a group together and level-up as a team. Learn stuff and tell people about it.

Use your writing skills to make existing documentation more readable or easier to understand

Break out those GitHub skills and make a pull request to improve documentation. Find something missing or lacking? Fill in that blank with your awesome technical writing.

Teach something you’ve learned about the project

Document your learning as if you are teaching it to someone else.

Submit issues when you come across bugs or errors

Find out how your project handles bug reports and submit them as needed. Don’t underestimate how helpful this is!

Become an ambassador

Find out how to become a champion for your project and community. Some projects have formal ambassador roles. If you’re excited about the work your community is doing, help spread the word.

Docs: Enable Pipeline Templates

Mon, 01 Jan 0001 00:00:00 +0000

hal config features edit --pipeline-templates true
hal deploy apply

Docs: Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

This means that we support only EC2 and Titus, and that we have limited feature and UI support. This guide walks through onboarding to Managed Delivery assuming that you are using EC2 or Titus.

Intro

To get started with Managed Delivery you’ll need an idea of what infrastructure you want to manage. You can pick a new piece of infrastructure that you want to create (a new load balancer, for example) or migrate an existing piece of infrastructure (an existing cluster, for example).

We recommend creating a new security group to test things out. This guide will walk you through creating a new managed security group.

Here are the steps you’ll complete:

First, you’ll create a YAML file to store your delivery config in.
Then, you’ll set up a Spinnaker pipeline to submit that YAML file to Spinnaker.
Finally, you’ll watch Spinnaker take actions to make your resource exist.

Background

Right now, most users manage infrastructure resources in the Spinnaker UI. These resources are security groups, load balancers, and clusters. To create these resources you manually define them by clicking the create button in the Spinnaker UI.

Managed Delivery changes this interaction. To define resources that Spinnaker manages, you’ll create a YAML-based “delivery config” file specifying the desired state of your infrastructure resources. “Desired state” is a new concept in Spinnaker that means you’re specifying what you want your infrastructure to always look like, regardless of how Spinnaker decides to make it look that way. In practice, this desired state will often look like the configuration options you’re used to defining through the Spinnaker UI.

To learn more about delivery configs, check out the Delivery Configs docs

Once you create a YAML delivery config, you’ll need to tell Spinnaker to manage it. This is done by submitting the file to Spinnaker’s API. You can submit a delivery config from your local machine, but that’s not very traceable or repeatable.

We recommend checking delivery configs into your git repository in a .spinnaker/ directory at the base level of your repository. In order to get Spinnaker to pick up your configuration (and monitor for changes) you should create a git-triggered pipeline that submits those files to Spinnaker.

An important note: Spinnaker is the source of truth for desired state of resources. You must submit your YAML files to Spinnaker every time they change.

All the instructions for this flow are captured in this doc.

Let’s get started!

Prerequisites

Spinnaker takes a lot of actions on your behalf, and we do that by using the permissions of a service account. Make sure you have one set up before you continue.

Create Your First Resource

We’re going to create a test security group. The new security group will be called YOUR_APP-md. For example, if the app that you’re using is keeldemo then the security group will be keeldemo-md.

We’re going to base this security group on your existing application security group. If you want to, locate your application security group in the Spinnaker UI and inspect the rules you have on it. This security group is usually auto-created for you and is identical to the name of your app.

To create a resource definition based on this security group, we’re going to use the Spinnaker API to export your existing application security group to a YAML definition (Note: this will have UI support in the future).

The base URL for exporting a security group is:

/managed/resources/export/aws/AWS_ACCOUNT_NAME/securitygroup/APP_NAME_OR_SG_NAME?serviceAccount=SERVICE_ACCOUNT

Replace AWS_ACCOUNT_NAME with the account your app is in. You can check the credentials endpoint in Spinnaker API for a list of valid account names.
Replace APP_NAME_OR_SG_NAME with either the name of your application or the name of the security group you want to export. You probably have a security group with the same name as your app, which is the one you should use here.
Replace SERVICE_ACCOUNT with the same service account you created or found in the prerequisites step.

If you’ve gotten all the information right a resource configuration will show up. See the API docs for more information on usage.

Save Your Resource In Your Git Repository

The next step in managing a resource is to check it into a git repository. Create a .spinnaker/ directory in the git repository of your application. This should be at the top level of your repo. In this directory, create a file called spinnaker.yml (your delivery config) and paste in this skeleton delivery config:

name: a-manifest-name # change me
application: yourapp # change me
artifacts: []
environments:
  - name: testing # this is a fine default but you can change it
    constraints: []
    notifications: []
    resources: []

Be sure to change the manifest name (a human readable name) and the application to your app. You can also change the environment name.

Next, we will add the security group to the list of resources.

Paste in the YAML output you got from the export endpoint in the previous step into the list of resources. It’ll look something like this:

name: a-manifest-name
application: yourapp
artifacts: []
environments:
  - name: testing
    constraints: []
    notifications: []
    resources: # you can add more resources here, this is a list
    - apiVersion: ec2.spinnaker.netflix.com/v1
      kind: security-group
      # ... the other info here from the security group you just exported

Next, modify the resource you just added to have a new name! Edit the moniker section of the resource you just pasted:

...
    moniker:
      app: YOUR_APP_NAME
      stack: md
...

This will name your security group YOUR_APP_NAME-md. For example, my sample security group for my app keeldemo will have a moniker section like:

...
    moniker:
      app: keeldemo
      stack: md
...

which will name the security group keeldemo-md.

:stop_sign: STOP! Don’t commit your delivery config yet! Just save it locally.

Repository Permissions

Make sure your git repository is accessible by Spinnaker.

Setting Up Resource Submission

Since Spinnaker is the source of truth (and not Git) you need to submit your managed resource configuration to Spinnaker every time there are changes. There’s a Spinnaker stage for this! See the docs on Git-based-workflows for detailed instructions.

Testing Out Resource Submission

Now that you have a pipeline in your application that’s triggered off of a git commit to your repository, you can commit your first resource configuration file. Commit the file you created earlier (.spinnaker/spinnaker.yml) and push it to the master branch of your repository.

Iterating on Managed Configurations

It’s hard to get YAML right every time. We have an endpoint where you can check if your YAML is right and see if there is a diff between your desired state (in your YAML) and the current state of the resource.

In the swagger api view of your Spinnaker instance, scroll to the managed-controller and find the /managed/delivery-configs/diff endpoint.

Paste in your delivery config YAML file, choose application/x-yaml for the content type, and hit execute. If the resource YAML has an invalid format, you’ll get an error back with the reason why. If the resource YAML is correct, you’ll see some information about whether or not there is a diff, and what that diff is.

For more help with this endpoint, see the the API doc.

Is Anything Happening?

You can see the actions Spinnaker is taking on your behalf in the Tasks view for your app.

If you navigate to Security Groups you should be able to see a small flag on the resource that will show you it’s being manged. If you click on a managed resource you will see the available managed actions.

Under the actions you can click to see the resource definition that Spinnaker has for that resource. You can also see the history of that resource. The history view shows every diff Spinnaker has seen, and every action taken to resolve that diff.

Find Your Resource ID

For some API calls you need the resource ID. You can find this by clicking on the resource in the UI, and then clicking on Resource Actions -> Raw Source (right side on the panel that pops out). The ID will be in the the metadata section and will be human readable, like ec2:cluster:test:keeldemo-main.

Pausing Management

Sometimes you need to take a break. Maybe there’s a problem you’re trying to diagnose and you need Spinnaker to stop taking actions on a resource. Maybe there’s a bug and it’s preventing you from using Spinnaker to manage resources. That’s ok! It’s easy to pause management.

Navigate to your application in Spinnaker and click on the CONFIG tab in the top right. If you have 1 or more managed resources, a section will appear called Managed Resources. That section contains a pause/resume button which you can use to stop and start management of your Resources.

Cleaning Up / Opting Out

If you don’t want Spinnaker to manage your resource declaratively anymore you can delete the declarative config from Spinnaker:

DELETE /managed/delivery-configs/{deliveryConfigName}

This endpoint is available in the Swagger UI as well.

This will not delete the underlying infrastructure that was previously in your delivery config. It will only delete the managed definition and stop Spinnaker from taking action. Use the Spinnaker UI to delete any resources you created.

Next Steps

Hooray! You created your first managed resource. If you’d like, you can follow these same steps to create clusters, load balancers, and security groups for your application. You can add resources to the environment you’ve created by added them to the list of resources.

Docs: API

Mon, 01 Jan 0001 00:00:00 +0000

The endpoints for managed delivery are located in the managed-controller. The endpoints are easily accessible from the swagger api page of your Spinnaker instance.

This doc provides samples for how to use the endpoints.

Export an Existing Resource

The export endpoint (under the “managed-controller”) allows you to take an existing resource and export it to a yaml config.

GET /managed/resources/export/{cloudProvider}/{account}/{type}/{name}?serviceAccount=yourServiceAccount@company.com

These definitions can be stored in your git repo.

Validating yaml

The ad-hoc diff endpoint allows you to see if a resource config is valid, and to see the difference between config (desired state) and reality.

POST /managed/resources/diff -d "{YOUR_RESOURCE_CONFIG}"

If the resource configuration is valid you’ll see information returned about the resource, like whether or not there is a diff, what the diff is, and what the resource name will be. If the resource configuration is invalid you’ll see an error and some text indicating what is wrong about the schema.

Viewing a Resource

The UI will show a flag on each resource that is declaratively managed. If you click on the resource, you can view its definition (raw source)

If you’d rather hit the api directly, you can refer to the UI for the resource name, or the logs for the publish stage.

GET /managed/resources/{name}

Viewing a Resource History

Spinnaker will take actions to make sure that your resource matches what you’ve defined. To view those actions in a list you can hit the history endpoint:

GET /history/{name}

This endpoint shows you why an action was taken on a resource, and what that action was. Soon there will be a nicer UI for this.

Docs: Managed Delivery Artifacts

Mon, 01 Jan 0001 00:00:00 +0000

Managed Delivery supports two types of delivery artifacts: debians and Docker images.

Delivery artifacts are meant to provide Spinnaker information about:

Where to locate the available versions of an artifact you care about
How to choose the latest version from a list of versions

Spinnaker needs this information to make decisions about when and how to roll out new versions as they become available. Artifacts are defined with name, type, and version strategy information.

Debians

** Warning: this is Netflix specific. Interested in contributing? Reach out to us in the #sig-spinnaker-as-code slack channel. **

All debians at Neflix are named and compared in the same way. For example, a debian might look like this:

# package name is keel
keel_0.353.0-h290.ae13adb_all.deb

We compare the semantic version located after the packagename_ (keel_ in this example) and before the -h.

Debians have status information on them. A debian can have any of these statuses:

snapshot
candidate
release

which are parsed from the semantic version of the debian.

To define a debian artifact for use with your delivery config, you’ll do something like this:

artifacts:
- name: keeldemo
  type: deb
  statuses: ["RELEASE"] # This is optional
  reference: my-debian-artifact # optional human-readable reference to be used elsewhere in the config, defaults to artifact name

If you provide status information, you limit the possible artifacts that can be deployed to your environment. This is useful if you have a test environment that only snapshots should be deployed to, or a prod environment that only release artifacts should be deployed to.

Status information is optional. If you leave it out, versions with any status will be considered eligible for deployment into your environment.

Docker Images

Docker versioning is denoted by the tag value. At Netflix, there are two main ways versions are structured:

# branch-jenkinsJob.commitSha
master-h5.5a52206

# semver-jenkinsJob.commitSha
v1.12.1-h1159.b839a00

Basic Configuration

In order to support the different ways of doing versioning, docker artifacts have some more config options. The basic structure is like this:

artifacts:
- name: emburns/spin-titus-demo
  type: docker
  reference: my-docker-artifact # optional human-readable reference to be used elsewhere in the config, defaults to artifact name
  tagVersionStrategy: branch-job-commit-by-job

Note that just like with pipelines, we don’t support using a constantly updated latest tag.

The key difference here is the tagVersionStrategy field, which indicates how we should sort a list of tags to choose the most recent software version.

The tagVersionStrategy Options:

increasing-tag: your tags are all integers, and we should choose the highest integer as the latest
semver-tag: your tags are a single semantic version (with or without a v prefix), and we should choose the largest semantic version
branch-job-commit-by-job: your tags are of the format “branch-jenkinsJob.commitSha” (like master-h5.5a52206), and we should choose the latest based on the highest jenkins job number (h5 in this example)
semver-job-commit-by-job: your tags are of the format “semver-jenkinsJob.commitSha” (like v1.12.1-h1159.b839a00), and we should choose the latest based on the highest jenkins job number (h1159 in this example)
semver-job-commit-by-semver: your tags are of the format “semver-jenkinsJob.commitSha” (like v1.12.1-h1159.b839a00), and we should choose the latest based on the largest semver (v1.12.1 in this example)

Advanced Configuration

If none of these work for you, you can provide your own regex to capture something to compare (using a capture group ). You must create a regex that parses valid tags and captures a string from them which can be compared either as an increasing number or a semantic version. The capture group must only capture one thing (you can’t have multiple capture groups).

For example, let’s say your tags look like master-h5.5a52206 and we didn’t have a built in strategy for this style. You’d define an artifact like this:

artifacts:
- name: emburns/spin-titus-demo
  type: docker
  reference: my-docker-artifact # optional human-readable reference to be used elsewhere in the config, defaults to artifact name
  tagVersionStrategy: increasing-tag
  captureGroupRegex: ^master-h(\d+).*$

The regex capture group (the parens in the regex expression) indicates that out of the string tag we should grab any numbers after the h and before the .. The version strategy indicates that we will take the result of the capture group and compare it as an increasing number.